Pith. sign in

REVIEW 4 major objections 5 minor 106 references

Extended Version: It Should Be Easy but... New Users Experiences and Challenges with Secret Management Tools

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that new users of secret management tools fail not because the tools are hard to use but because the official documentation omits key CLI details, lacks language-specific examples such as Python, and describes commands inc

desk verdict Solid exploratory qualitative study with a fixable statistical error and an overstated causal claim; worth refereeing. read the letter →

arxiv 2509.09036 v1 pith:MCR7C7ER submitted 2025-09-10 cs.HC

Lorenzo Neil , Deepthi Mungara , Laurie Williams , Yasemin Acar , Bradley Reaves This is my paper · ORCID
classification cs.HC
keywords secretmanagementtoolsusablesecuritytooldocumentationnewuseronboardingCLIdeveloperadoptionqualitativestudyhard-codedsecrets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Secret management tools (SMTs) protect API keys and passwords by storing them centrally, yet developers keep leaking secrets and report that learning these tools is hard. This paper argues that the main barrier is not tool design but the official documentation: in a two-part lab study, 21 users with prior secret-handling experience removed a hard-coded token, stored it in one of three cross-cloud SMTs, and injected it into a Python app from the command line. Participants praised the tools' dashboards, flexibility, and copy-to-clipboard conveniences, but two-thirds of them abandoned the official docs at some point and the number who did doubled on the injection task. The sticking points were concrete: missing explanations for CLI flags and arguments, no Python example for the key 'run' command, and inconsistent command descriptions across pages. If those documentation gaps are closed, the paper argues, new users would be less likely to fall back on forums and unofficial workarounds, making secure secret management a realistic onboarding task.

What carries the argument

Two-task observation protocol with deviation tracking. Task 1 exercises secret storage and access; Task 2 exercises secret injection via a tool's 'run' command, with each participant assigned to one of three cross-cloud SMTs. The paper records two behaviors as objective markers of documentation failure—'usage of secondary sources' (looking outside official docs) and 'workaround attempts' (running non-recommended commands like exporting a secret in plaintext)—and uses post-task interviews to attribute those behaviors to specific documentation characteristics: missing flag/argument explanations, missing Python examples, and inconsistent command presentations across documentation pages.

What would settle it

Run the same two-task setup with two documentation arms: current official docs versus docs augmented with a Python example, a flag/argument glossary, and identical command descriptions across pages. If the augmented arm shows no reduction in secondary-source usage, workaround attempts, or completion time, the paper's central causal claim is wrong.

Watch

Extended reading notes

Core claim

The paper's central discovery is that documentation quality, not functionality, separates successful from struggling SMT onboarding. Task 1 (store and retrieve a secret) went smoothly for all 21 participants, but Task 2 (inject the secret into a local Python application with a single CLI run command) was completed by only 18 within the time limit, and 16 participants used a secondary source or tried a workaround. Participants who read the same command described differently in quick-start guides, reference pages, and dashboard hints lost time choosing between versions; participants who found only NPM or Flask examples could not tell whether the command applied to Python; and unexplained flags

Load-bearing premise

The study's load-bearing premise is that 21 graduate students with some prior secret-management experience represent the larger population of new developers who will first adopt SMTs; if that proxy fails, the observed difficulties and their attribution to documentation may not generalize.

Editorial extensions

If this is right

  • Fixing documentation content should be the first lever for SMT onboarding: adding explicit flag/argument explanations and a Python injection example should reduce deviations and completion times on a two-command workflow.
  • Consistency across documentation surfaces matters: quick-start guides, CLI reference pages, and in-dashboard hints that describe the same command differently are themselves a usability bug.
  • When official documentation fails, users do not simply give up; they turn to forums and invent commands, which can undercut the secure behavior the tool is meant to provide (e.g., printing secrets in plaintext).
  • Documentation search and navigation deserve investment on par with tool features: participants were slowed by search that returned LLM-style answers without links and by relevant help hidden in unexpected places like the web dashboard.
  • Since the study used already-experienced users and still found difficulty, real-world onboarding for less-prepared newcomers is likely to be worse; the paper positions its results as a best-case baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The pattern likely extends beyond SMTs: any security tool whose CLI docs omit language-specific examples and flag glossaries may spawn the same forum-and-workaround cycle, since the failure is in the learning resource, not the software.
  • The 'natural experiment' in the paper—one vendor added a Python example to its dashboard mid-study and participants who noticed it completed faster—suggests a cheap, testable fix: a single missing language example may be the highest-leverage documentation improvement.
  • Documentation failures may have direct security costs: workaround commands such as exporting secrets in plaintext or using the injection command to print secrets can themselves create leak opportunities, so doc quality is not just a usability issue.
  • A plausible extension is to test whether surfacing the relevant command inside the tool's own web UI (as one vendor did) outperforms burying it on a separate documentation page, since participants were surprised to find help in the dashboard.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a qualitative laboratory study of 21 Computer Science master's students, seven assigned to each of three cross-cloud secret management tools (Doppler, HashiCorp Vault Secrets, Infisical). Participants performed two tasks: first removing a hard-coded secret from a Python file, storing it in the SMT's web UI, and accessing it via the CLI; second, injecting the secret into the local Python application. After each task, participants were interviewed about their experiences with the tool, its documentation, and any secondary sources they consulted. The central finding is that participants generally appreciated the SMTs' functionality and interfaces, but documentation deficiencies—missing command-line argument details, lack of relevant Python examples, and inconsistent command presentations—were associated with increased difficulty, use of secondary sources, and workaround attempts. The paper reports that only 18 of 21 participants completed the injection task within the time limit, and that deviations from the documented path doubled from Task 1 to Task 2. It concludes with recommendations for SMT vendors about documentation content, structure, and supportive features.

Significance. The topic is timely and underexplored: secret leakage remains a serious problem, and SMT onboarding burdens have been reported but not closely observed. The study's strengths include a clear, reproducible protocol; a published replication package; triangulation of task performance, observed deviations, and interview data; and transparent inter-coder agreement (Krippendorff's alpha > 0.75). If the association between documentation quality and SMT usability holds, the paper provides concrete, actionable documentation problems (e.g., missing CLI argument explanations, lack of language-specific examples, inconsistent command listings) that SMT vendors can address. However, the small, homogeneous sample (n=7 per tool, all CS master's students from one university), between-subjects design, and non-manipulated documentation differences limit causal inference. The study is best read as an exploratory, hypothesis-generating qualitative investigation rather than a definitive causal demonstration.

major comments (4)
  1. [Section 4.1.1] The paper states, 'We found a statistically significant difference in distributions (H=6.50, P=0.39) with a "large" effect size (ε²=0.250).' As written, P=0.39 is not statistically significant and directly contradicts the preceding sentence. With df=2, H=6.50 corresponds to p≈0.039, so this is likely a typo, but it must be corrected. Given n=7 per group and the arbitrary assignment of 30 minutes for incomplete tasks, I recommend removing the phrase 'statistically significant' entirely or explicitly labeling the result as provisional and reporting the exact p-value (p=0.039 if that is the intended value).
  2. [Sections 3.2.3, 4.1.1, and Table 2] The 'natural experiment' invoked in Section 3.2.3 cuts against the causal claim that missing Python examples were a principal obstacle. In Task 2, Infisical—the only SMT without any Python injection example in its public documentation—had the fastest median completion time among completers (5.3 min), while HCP Vault Secrets, which had a dashboard Python example, had a 16.7 min median and high workaround use. The paper acknowledges low statistical power but does not reconcile this observation with the claim that documentation quality caused reduced effectiveness. Combined with the statement in the Introduction that 'our focus on it sharpened during the study,' the design cannot support a causal attribution. Please reframe the central claim as hypothesis-generating and emphasize that the evidence supports 'participants perceived documentation gaps and reported seeking other sources,' not t
  3. [Section 3.2.4 and RQ1 (Section 1)] The abstract and RQ1 claim that participants struggled to 'securely remediate a hard-coded secret,' but the recorded measures are task completion, completion time, secondary-source use, and workaround attempts. There is no independent check of whether the final SecureSecret.py still contained the hard-coded secret or whether the secret remained exposed in logs, shell history, or environment variables. For example, using the injection command to print a plaintext secret in Task 1 is classified as a workaround, but it may still remove the hard-coded secret. Thus the 'secure remediation' outcome is not directly measured. Either add a verification step that inspects the final code and runtime state, or narrow the claim to 'success in following the documented path' and 'perceived effectiveness.'
  4. [Section 3.5 (Limitations)] The limitations section does not address the retrospective sharpening of the documentation focus (Section 3.2.3) or the absence of a control condition. The interview guide asks specifically about documentation content and structure after every task, which may prime participants to attribute difficulties to documentation. The study cannot separate documentation effects from task novelty, tool-specific UI differences, or individual prior experience. Please add these as explicit limitations and soften the corresponding causal language in the abstract and conclusion.
minor comments (5)
  1. [Section 5.1] Typographical errors: 'Taks 1' should be 'Task 1'; 'we did not not observe' should be 'we did not observe.'
  2. [Section 4.3.1] Typographical error: 'Inifisical' should be 'Infisical.'
  3. [Tables 1 and 2] The plain-text rendering of the tables makes it difficult to distinguish the 'Used Secondary Source' and 'W.A Method' checkmarks. Please ensure the final PDF table is unambiguous, especially since the text reports only two secondary-source users in Task 1 while the table rows appear to show more checkmarks in the corresponding column if read without the overall row.
  4. [Section 4.1.1] The Kruskal-Wallis test is described as performed 'out of curiosity' and 'should not be considered definitive.' Given the small sample and arbitrary censoring, consider moving this analysis to an appendix or reporting effect sizes with confidence intervals rather than presenting p-values in the main results.
  5. [Section 3.2.3] The timing of documentation updates is described in the methods section, but it is also a limitation of the natural experiment. Please state explicitly in the limitations section that HCP Vault Secrets added a Python example during the study and that Doppler's Python example is external to its main documentation, as this affects the interpretation of the tool comparisons.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claims are grounded in independent observations and interviews; cited prior work by the authors is background motivation, not load-bearing.

full rationale

The paper's central finding that perceived documentation gaps led new SMT users to seek secondary sources or workarounds and increased effort is derived from direct observation (task completion, secondary-source use, workaround attempts) and post-task interviews coded with intercoder agreement, not from a fitted parameter or from the authors' prior results. No equation or model is fitted, so no prediction reduces to an input by construction. The self-citations ([12,14] by Neil/Reaves/Williams and [53] with Acar) are used only to motivate the research gap and are not the evidence for the study's conclusions; the study independently reproduces the phenomenon in a controlled setting. The manuscript explicitly flags its own limitations: Section 3.5 states the sample is CS Master's students and may introduce confounding, and Section 3.2.3 acknowledges the 'natural experiment' in which HCP added a Python example after piloting and Infisical never did, while also reporting low statistical power. Those are correctness/generalizability caveats, not circularity. The strongest claim is appropriately hedged ('when they felt tool documentation was insufficient'), matching the subjective interview data. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling is present. Score 2 reflects only the presence of minor, non-load-bearing self-citations in the background.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters are fitted in this qualitative study. The main axioms are domain assumptions about participant truthfulness and task representativeness, both explicitly or implicitly relied upon throughout the methodology. No new entities, particles, or forces are introduced.

assumptions (2)
  • domain assumption Participants' self-reported experiences and challenges are reasonably accurate and not strongly biased by the laboratory setting or the researchers' expectations.
    The study relies on interview self-reports after each task to attribute difficulties to documentation; demand characteristics could inflate these reports. See Sections 3.3 and 4.3.
  • domain assumption The two chosen tasks (store/access, then inject) are representative of real-world onboarding workflows for secret management tools.
    The authors design tasks based on OWASP guidance and prior workflows and assume success on these tasks indicates effective use in practice. See Section 3.2.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extended Version: It Should Be Easy but... New Users Experiences and Challenges with Secret Management Tools." pith.science (2026). https://pith.science/paper/MCR7C7ER

@misc{pith2026250909036,
  author       = {Pith},
  title        = {Pith review of: Extended Version: It Should Be Easy but... New Users Experiences and Challenges with Secret Management Tools},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MCR7C7ER}},
  note         = {Machine review of arXiv:2509.09036}
}
read the original abstract

Software developers face risks of leaking their software secrets, such as API keys or passwords, which can result in significant harm. Secret management tools (SMTs), such as HashiCorp Vault Secrets or Infisical, are highly recommended by industry, academia, and security guidelines to manage secrets securely. SMTs are designed to help developers secure their secrets in a central location, yet secrets leaks are still commonplace, and developers report difficulty in learning how to setup and use SMTs. While SMTs typically come with publicly available help resources (e.g., tool documentation and interfaces), it is unclear if these actually help developers learn to effectively use SMTs. Without usable help resources that onboards developers, quick adoption and effective use of SMTs may be unrealistic. In a qualitative two-step study, we observed 21 new users in person while they used SMTs to perform two secret management tasks: secret storage and access, then secret injection. We interviewed participants after each task to identify their challenges and experiences using SMTs, with the assistance of help resources. While our study sample is narrow, it serves as a reasonable proxy for new developers who are likely to adopt SMTs early in their careers. We found that even in a laboratory setting where new users found tool functionality, interface flexibility helpful, they still experienced increased difficulty to effectively use SMTs to securely remediate a hard-coded secret when they felt tool documentation was insufficient and it motivated participants to deviate from official tool documentation to access secondary sources or attempt workaround methods. Specific challenges reported by participants were tool documentation content quality, navigation difficulties with both tool documentation and web interfaces for finding helpful content, and supportive tool features.

Figures

Figures reproduced from arXiv: 2509.09036 by the authors.

Figure 3
Figure 3. Sample presentation of the list of CLI commands [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 5
Figure 5. Additional part of HashiCorp vault secrets docu [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

106 extracted references · 1 canonical work pages

  1. [1]

    Lawrence Abrams. 2024. New York Times source code stolen using exposed GitHub token. https://www.bleepingcomputer.com/news/security/new-york- times-source-code-stolen-using-exposed-github-token/. Accessed: 2024-00-00

    2024

  2. [2]

    Yasemin Acar, Michael Backes, Sascha Fahl, Simson Garfinkel, Doowon Kim, Michelle L Mazurek, and Christian Stransky. 2017. Comparing the usability of cryptographic apis. In2017 IEEE Symposium on Security and Privacy (SP). IEEE, 154–171

    2017

  3. [3]

    Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L Mazurek, and Christian Stransky. 2016. You get where you’re looking for: The impact of information sources on code security. In2016 IEEE Symposium on Security and Privacy (SP). IEEE, 289–305

    2016

  4. [4]

    Yasemin Acar, Christian Stransky, Dominik Wermke, Charles Weir, Michelle L Mazurek, and Sascha Fahl. 2017. Developers need support, too: A survey of security advice for software developers. In2017 IEEE Cybersecurity Development (SecDev). IEEE, 22–26. CCS ’25, October 13–17, 2025, Taipei, Taiwan Lorenzo Neil et al

    2017

  5. [5]

    Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C Shepherd. 2020. Software documentation: the practitioners’ perspective. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 590–601

    2020

  6. [6]

    Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares- Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documentation issues unveiled. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1199–1210

    2019

  7. [7]

    Deeksha M Arya, Jin LC Guo, and Martin P Robillard. 2023. How programmers find online learning resources.Empirical Software Engineering28, 2 (2023), 23

    2023

  8. [8]

    Deeksha M Arya, Jin LC Guo, and Martin P Robillard. 2024. Why People Con- tribute Software Documentation. InProceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engi- neering. 91–96

    2024

Show all 106 references
  1. [9]

    AWS. 2024. AWS Key Management Service. https://aws.amazon.com/kms. Accessed: 2024-00-00

    2024

  2. [10]

    Wei Bai, Moses Namara, Yichen Qian, Patrick Gage Kelley, Michelle L Mazurek, and Doowon Kim. 2016. An Inconvenient Trust: User Attitudes toward Security and Usability Tradeoffs for {Key-Directory} Encryption Systems. InTwelfth Symposium on Usable Privacy and Security (SOUPS 20...

    2016

  3. [11]

    Sebastian Baltes, Christoph Treude, and Martin P Robillard. 2020. Contextual documentation referencing on stack overflow.IEEE Transactions on Software Engineering48, 1 (2020), 135–149

    2020

  4. [12]

    Setu Kumar Basak, Lorenzo Neil, Bradley Reaves, and Laurie Williams. 2022. What are the practices for secret management in software artifacts?. In2022 IEEE Secure Development Conference (SecDev). IEEE, 69–76

    2022

  5. [13]

    Setu Kumar Basak, Lorenzo Neil, Bradley Reaves, and Laurie Williams. 2023. SecretBench: A Dataset of Software Secrets.arXiv preprint arXiv:2303.06729 (2023)

    2023 arXiv

  6. [14]

    Setu Kumar Basak, Lorenzo Neil, Bradley Reaves, and Laurie Williams. 2023. What Challenges Do Developers Face About Checked-in Secrets in Software Artifacts?arXiv preprint arXiv:2301.12377(2023)

    2023 arXiv

  7. [15]

    Supraja Baskaran, Lianying Zhao, Mohammad Mannan, and Amr Youssef. 2023. Measuring the leakage and exploitability of authentication secrets in super-apps: The wechat case. InProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses. 727–743

    2023

  8. [16]

    Avinash Bhat, Disha Shrivastava, and Jin LC Guo. 2024. Do LLMs meet the needs of software tutorial writers? Opportunities and design implications. In Proceedings of the 2024 ACM Designing Interactive Systems Conference. 1760– 1773

    2024

  9. [17]

    Tricks of the trade

    Bjørnar Blaalid. 2018. “Tricks of the trade” – The art and method of combin- ing interviews and participating observations to generate data on drug users participating in rehabilitation programs.ResearchGate(04 2018), 13

    2018

  10. [18]

    Raluca Budiu. 2024. Between-Subjects vs. Within-Subjects Study Design. https: //www.nngroup.com/articles/between-within-subjects/. Accessed: 2024-00-00

    2024

  11. [19]

    Matt Burgees. 2024. Thousands of Corporate Secrets Were Left Exposed. This Guy Found Them All. https://www.wired.com/story/secret-hunting-bill- demirkapi/. Accessed: 2024-00-00

    2024

  12. [20]

    Stephen Cass. 2024. The Top Programming Languages 2024. https://spectrum. ieee.org/top-programming-languages-2024. Accessed: 2024-00-00

    2024

  13. [21]

    Ramaswamy Chandramouli. 2024. Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD pipelines. https://csrc.nist.gov/ pubs/sp/800/204/d/ipd?ref=blog.gitguardian.com. Accessed: 2024-00-00

    2024

  14. [22]

    Jie-Cherng Chen and Sun-Jen Huang. 2009. An empirical analysis of the impact of software development problem factors on software maintainability.Journal of Systems and Software82, 6 (2009), 981–992

    2009

  15. [23]

    Lan Cheng, Emerson Murphy-Hill, Mark Canning, Ciera Jaspan, Collin Green, Andrea Knight, Nan Zhang, and Elizabeth Kammer. 2022. What improves devel- oper productivity at google? code quality. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symp...

    2022

  16. [24]

    Dev Community. 2024. Build and deploy a web app with Python, Flask, and Doppler. https://dev.to/lordghostx/build-and-deploy-a-web-app-with-python- flask-and-doppler-9jm. Accessed: 2024-00-00

    2024

  17. [25]

    Markus Dahlmanns, Constantin Sander, Robin Decker, Klaus Wehrle, Jan Pen- nekamp, Anastasiia Belova, Thomas Bergs, Matthias Bodenbenner, Andreas Bührig-Polaczek, Ike Kunze, et al. 2023. Secrets Revealed in Container Images: An Internet-wide Study on Occurrence and Impact. InAC...

    2023

  18. [26]

    Doppler. 2024. The New Era of Secrets Management. https://www.doppler.com/. Accessed: 2024-00-00

    2024

  19. [27]

    Doppler. 2024. Python. https://docs.doppler.com/docs/vscode-python. Accessed: 2024-00-00

    2024

  20. [28]

    Neil A Ernst and Martin P Robillard. 2023. A study of documentation for software architecture.Empirical Software Engineering28, 5 (2023), 122

    2023

  21. [29]

    Hugging Face. 2024. openai/whisper-medium. https://huggingface.co/openai/ whisper-medium. Accessed: 2024-00-00

    2024

  22. [30]

    These results must be false

    Marcel Fourné, Daniel De Almeida Braga, Jan Jancar, Mohamed Sabt, Peter Schwabe, Gilles Barthe, Pierre-Alain Fouque, and Yasemin Acar. 2024. “These results must be false”: A usability evaluation of constant-time analysis tools. In 33th USENIX Security Symposium (USENIX Security 2024)

    2024

  23. [31]

    2024.ReCal2: Reliability for 2 Coders

    Deen Freelon. 2024.ReCal2: Reliability for 2 Coders. Accessed: 2024-00-00

    2024

  24. [32]

    Deen Freelon. 2024. ReCal3: Reliability for 3+ Coders. http://dfreelon.org/utils/ recalfront/recal3/. Accessed: 2024-00-00

    2024

  25. [33]

    Peggy Fung, Lam-for Kwok, and Dennis Longley. 2003. Electronic information security documentation. InProceedings of the Australasian Information Security Workshop Conference on ACSW Frontiers 2003 - Volume 21(Adelaide, Australia) (ACSW Frontiers ’03). Australian Computer Socie...

    2003

  26. [34]

    g2. 2024. Best Secrets Management Tools". https://www.g2.com/categories/ secrets-management-tools. Accessed: 2024-00-00

    2024

  27. [35]

    GitHub. 2024. Managing your account-specific secrets for GitHub Codespaces. https://docs.github.com/en/codespaces/managing-your- codespaces/managing-your-account-specific-secrets-for-github-codespaces. Accessed: 2024-00-00

    2024

  28. [36]

    GitHub. 2024. Using secrets in GitHub Actions. https://docs.github.com/en/ actions/security-for-github-actions/security-guides/using-secrets-in-github- actions. Accessed: 2024-00-00

    2024

  29. [37]

    Google. 2024. Google Secret Manager. https://cloud.google.com/secret-manager. Accessed: 2024-00-00

    2024

  30. [38]

    Peter Leo Gorski, Luigi Lo Iacono, Dominik Wermke, Christian Stransky, Sebas- tian Möller, Yasemin Acar, and Sascha Fahl. 2018. Developers deserve security warnings, too: On the effect of integrated security advice on cryptographic {API} misuse. InFourteenth Symposium on Usabl...

    2018

  31. [39]

    I just looked for the solution!

    Peter Leo Gorski, Sebastian Möller, Stephan Wiefling, and Luigi Lo Iacono. 2022. “I just looked for the solution!”On Integrating Security-Relevant Information in Non-Security API Documentation to Support Secure Coding Practices.IEEE Transactions on Software Engineering48, 9 (2...

    2022

  32. [40]

    HashiCorp. 2024. hcp vault-secrets run. https://developer.hashicorp.com/hcp/ docs/cli/commands/vault-secrets/run. Accessed: 2024-00-00

    2024

  33. [41]

    HashiCorp. 2024. What is HCP Vault Secrets? https://developer.hashicorp.com/ hcp/docs/vault-secrets/. Accessed: 2024-00-00

    2024

  34. [42]

    Andrew Head, Caitlin Sadowski, Emerson Murphy-Hill, and Andrea Knight

  35. [43]

    Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Rat- ner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2023. Tool documenta- tion enables zero-shot tool-usage with large language models.arXiv preprint arXiv:2308.00675(2023)

    2023 arXiv

  36. [44]

    Yizhan Huang, Yichen Li, Weibin Wu, Jianping Zhang, and Michael R Lyu. 2024. Your code secret belongs to me: neural code completion tools can memorize hard-coded credentials.Proceedings of the ACM on Software Engineering1, FSE (2024), 2515–2537

    2024

  37. [45]

    Soumya Indela, Mukul Kulkarni, Kartik Nayak, and Tudor Dumitraş. 2016. Help- ing Johnny encrypt: Toward semantic interfaces for cryptographic frameworks. InProceedings of the 2016 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Softw...

    2016

  38. [46]

    Infisical. 2024. Open Source Secret Management. https://infisical.com/. Accessed: 2024-00-00

    2024

  39. [47]

    itnews. 2024. AWS urges developers to scrub GitHub of secret keys. https://www.itnews.com.au/news/aws-urges-developers-to-scrub- github-of-secret-keys-375785. Accessed: 2024-00-00

    2024

  40. [48]

    Mackenzie Jackson. 2024. 8.5% of Docker Images Expose API and Private Keys. https://blog.gitguardian.com/8docker-images-api-and-private-keys/. Accessed: 2024-00-00

    2024

  41. [49]

    Shazia Jamshed. 2014. Qualitative research method-interviewing and observa- tion.Journal of Basic and Clinical Pharmacy05, 04 (2014), 87–88

    2014

  42. [50]

    Paul Jansen. 2024. TIOBE Index for November 2024. https://www.tiobe.com/ tiobe-index/. Accessed: 2024-00-00

    2024

  43. [51]

    Johannes Kaiser and Martin Reichenbach. 2002. Evaluating security tools to- wards usable security: A usability taxonomy for the evaluation of security tools based on a categorization of user errors. InIFIP World Computer Congress, TC

    2002

  44. [52]

    Eyal Katz. 2024. 5 Ways to Prevent Secrets Sprawl. https://spectralops.io/blog/5- ways-to-prevent-secrets-sprawl/. Accessed: 2024-00-00

    2024

  45. [53]

    Alexander Krause, Jan H Klemmer, Nicolas Huaman, Dominik Wermke, Yasemin Acar, and Sascha Fahl. 2023. Pushed by Accident: A {Mixed-Methods} Study on Strategies of Handling Secret Information in Source Code Repositories. In 32nd USENIX Security Symposium (USENIX Security 23). 2527–2544

    2023

  46. [54]

    Stefan Krüger, Sarah Nadi, Michael Reif, Karim Ali, Mira Mezini, Eric Bod- den, Florian Göpfert, Felix Günther, Christian Weinert, Daniel Demmler, et al

  47. [55]

    Stefan Krüger, Michael Reif, Anna-Katharina Wickert, Sarah Nadi, Karim Ali, Eric Bodden, Yasemin Acar, Mira Mezini, and Sascha Fahl. 2023. Securing your crypto-api usage through tool support-A usability study. In2023 IEEE Secure Development Conference (SecDev). IEEE, 14–25

    2023

  48. [56]

    Timothy C Lethbridge, Janice Singer, and Andrew Forward. 2003. How software engineers use documentation: The state of the practice.IEEE software20, 6 (2003), 35–39

    2003

  49. [57]

    Songsong Liu, Shu Wang, and Kun Sun. 2024. Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional Videos. Proceedings of the ACM on Human-Computer Interaction8, EICS (2024), 1–19

    2024

  50. [58]

    Nikolaos Lykousas and Constantinos Patsakis. 2024. Decoding developer pass- word patterns: A comparative analysis of password extraction and selection practices.Computers & Security145 (2024), 103974

    2024

  51. [59]

    Vlad Matsiiako. 2024. Top-10 Secret Management Tools in 2024. https://infisical. com/blog/best-secret-management-tools. Accessed: 2024-00-00

    2024

  52. [60]

    Dwayne Mcdaniel. 2024. A look at the future of supply chain and national security: Updates from CISA and NIST. https://blog.gitguardian.com/software- supply-chain-security-updates-from-cisa-and-nist/. Accessed: 2024-00-00

    2024

  53. [61]

    Dwayne Mcdaniel. 2024. Toyota Suffered a Data Breach by Accidentally Ex- posing A Secret Key Publicly On GitHub. https://blog.gitguardian.com/toyota- accidently-exposed-a-secret-key-publicly-on-github-for-five-years/. Accessed: 2024-00-00

    2024

  54. [62]

    Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice.Proceedings of the ACM on human-computer interaction3, CSCW (2019), 1–23

    2019

  55. [63]

    Medium. 2024. Injecting secrets to Kubernetes containers from the Doppler secrets manager. https://medium.com/@peterkracik/injecting-secrets-to- kubernetes-containers-from-the-doppler-secrets-manager-ef491a20f45b. Ac- cessed: 2024-00-00

    2024

  56. [64]

    Michael Meli, Matthew R McNiece, and Bradley Reaves. 2019. How bad can it git? characterizing secret leakage in public github repositories.. InNDSS

    2019

  57. [65]

    Microsoft. 2024. Azure Key Vault. https://learn.microsoft.com/en-us/azure/key- vault/. Accessed: 2024-00-00

    2024

  58. [66]

    Microsoft. 2024. Visual Studio Code. https://code.visualstudio.com/. Accessed: 2024-00-00

    2024

  59. [67]

    Justin Middleton, Emerson Murphy-Hill, and Kathryn T Stolee. 2020. Data analysts and their software practices: A profile of the sabermetrics community and beyond.Proceedings of the ACM on Human-Computer Interaction4, CSCW1 (2020), 1–27

    2020

  60. [68]

    Emerson Murphy-Hill, Ciera Jaspan, Caitlin Sadowski, David Shepherd, Michael Phillips, Collin Winter, Andrea Knight, Edward Smith, and Matthew Jorde. 2019. What predicts software developers’ productivity?IEEE Transactions on Software Engineering47, 3 (2019), 582–594

    2019

  61. [69]

    Olha Mykhaylova, Taras Fedynyshyn, and Artem Platonenko. 2024. Hardcoded credentials in Android apps: Service exposure and category-based vulnerability analysis.Cybersecurity Providing in Information and Telecommunication Systems II 20243826 (2024), 206–211

    2024

  62. [70]

    Mathieu Nassif, Alexa Hernandez, Ashvitha Sridharan, and Martin P Robillard

  63. [71]

    Mathieu Nassif, Zara Horlacher, and Martin P Robillard. 2022. Casdoc: unob- trusive explanations in code examples. InProceedings of the 30th IEEE/ACM international conference on program comprehension. 631–635

    2022

  64. [72]

    Mathieu Nassif and Martin P Robillard. 2023. A Field Study of Developer Documentation Format. InExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–7

    2023

  65. [73]

    Mathieu Nassif and Martin P Robillard. 2023. Identifying Concepts in Software Projects.IEEE Transactions on Software Engineering49, 7 (2023), 3660–3674

    2023

  66. [74]

    Mathieu Nassif and Martin P Robillard. 2023. Non Linear Software Documenta- tion with Interactive Code Examples.arXiv preprint arXiv:2311.18057(2023)

    2023 arXiv

  67. [75]

    Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Greiler. 2023. DevEx: What Actually Drives Productivity: The developer-centric approach to measuring and improving productivity.Queue21, 2 (2023), 35–53

    2023

  68. [76]

    Sanghak Oh, Kiho Lee, Seonhye Park, Doowon Kim, and Hyoungshick Kim

  69. [77]

    OWASP Cheat Sheet Series. 2025. Secrets Management Cheat Sheet. https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_ Cheat_Sheet.html. https://cheatsheetseries.owasp.org/cheatsheets/Secrets_ Management_Cheat_Sheet.html Accessed: 2025-07-23

    2025

  70. [78]

    Chris Parnin, Christoph Treude, Lars Grammel, and Margaret-Anne Storey

  71. [79]

    Reinhold Plösch, Andreas Dautovic, and Matthias Saft. 2014. The Value of Soft- ware Documentation Quality. In2014 14th International Conference on Quality Software. 333–342. doi:10.1109/QSIC.2014.22

    2014 doi

  72. [80]

    PyPI. 2024. doppler-env 0.3.1. https://pypi.org/project/doppler-env/. Accessed: 2024-00-00

    2024

  73. [81]

    Qualtrics. 2024. qualtrics. https://www.qualtrics.com/. Accessed: 2024-00-00

    2024

  74. [82]

    ReversingLabs. 2024. Secrets Exposed: How to mitigate risk from secrets leaks — and prevent future breaches. https://www.reversinglabs.com/blog/secure- your-development-secrets-3-essential-steps. Accessed: 2024-00-00

    2024

  75. [83]

    Martin P Robillard. 2009. What makes APIs hard to learn? Answers from developers.IEEE software26, 6 (2009), 27–34

    2009

  76. [84]

    Martin P Robillard and Yam B Chhetri. 2015. Recommending reference API documentation.Empirical Software Engineering20, 6 (2015), 1558–1586

    2015

  77. [85]

    Derek Robinson, Neil A Ernst, Enrique Larios Vargas, and Margaret-Anne D Storey. 2022. Error identification strategies for Python Jupyter notebooks. In Proceedings of the 30th IEEE/ACM International Conference on Program Compre- hension. 253–263

    2022

  78. [86]

    Judith Segal. 2007. Some problems of professional end user developers. InIEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2007). IEEE, 111–118

    2007

  79. [87]

    Vibha Singhal Sinha, Diptikalyan Saha, Pankaj Dhoolia, Rohan Padhye, and Senthil Mani. 2015. Detecting and mitigating secret-key leaks in source code repositories. In2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 396–400

    2015

  80. [88]

    Justin Smith, Lisa Nguyen Quang Do, and Emerson Murphy-Hill. 2020. Why can’t johnny fix vulnerabilities: A usability evaluation of static analysis tools for security. InSixteenth Symposium on Usable Privacy and Security (SOUPS 2020). 221–238

    2020

  81. [89]

    SM Sohan, Frank Maurer, Craig Anslow, and Martin P Robillard. 2017. A study of the effectiveness of usage examples in REST API documentation. In2017 IEEE symposium on visual languages and human-centric computing (VL/HCC). IEEE, 53–61

    2017

  82. [90]

    Prakash Somasundaram. 2024. Unified Secret Management Across Cloud Plat- forms: A Strategy for Secure Credential Storage and Access.Int. J. Comput. Eng. Technol15 (2024), 5–12

    2024

  83. [91]

    Stackoverflow. 2024. 2024 Developer Survey. https://survey.stackoverflow.co/ 2024/technology#2-programming-scripting-and-markup-languages. Accessed: 2024-00-00

    2024

  84. [92]

    Margaret-Anne Storey, Daniel Russo, Nicole Novielli, Takashi Kobayashi, and Dong Wang. 2024. A disruptive research playbook for studying disruptive innovations.ACM Transactions on Software Engineering and Methodology(2024)

    2024

  85. [93]

    Margaret-Anne Storey, Thomas Zimmermann, Christian Bird, Jacek Czerwonka, Brendan Murphy, and Eirini Kalliamvakou. 2019. Towards a theory of software developer job satisfaction and perceived productivity.IEEE Transactions on Software Engineering47, 10 (2019), 2125–2142

    2019

  86. [94]

    Siddharth Subramanian, Laura Inozemtseva, and Reid Holmes. 2014. Live API documentation. InProceedings of the 36th international conference on software engineering. 643–652

    2014

  87. [95]

    Mohammad Tahaei and Kami Vaniea. 2019. A survey on developer-centred security. In2019 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 129–138

    2019

  88. [96]

    Christoph Treude and Martin P Robillard. 2016. Augmenting API documenta- tion with insights from stack overflow. InProceedings of the 38th International Conference on Software Engineering. 392–403

    2016

  89. [97]

    Gias Uddin and Martin P Robillard. 2015. How API documentation fails.Ieee software32, 4 (2015), 68–75

    2015

  90. [98]

    Stefan Wagner and Emerson Murphy-Hill. 2019. Factors that influence pro- ductivity: A checklist.Rethinking productivity in software engineering(2019), 69–84

    2019

  91. [99]

    Thomas Weber, Alois Zoitl, and Heinrich Hußmann. 2019. Usability of Devel- opment Tools: A CASE-Study. In2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). 228–235. doi:10.1109/MODELS-C.2019.00037

    2019

  92. [100]

    Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. 2019. A large- scale empirical study on code-comment inconsistencies. In2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 53–64

    2019

  93. [101]

    Jim Witschey, Olga Zielinska, Allaire Welk, Emerson Murphy-Hill, Chris May- horn, and Thomas Zimmermann. 2015. Quantifying developers’ adoption of security tools. InProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 260–271. A Interview Question...

    2015

  94. [2012]

    Rep11 (2012)

    Crowd documentation: Exploring the coverage and the dynamics of API discussions on Stack Overflow.Georgia Institute of Technology, Tech. Rep11 (2012)

    2012

  95. [2017]

    In2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)

    Cognicrypt: Supporting developers in using cryptography. In2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). Extended Version: It Should Be Easy but... New Users’ Experiences and Challenges with Secret Management Tools CCS ’25, October 13–17,...

    2025

  96. [2018]

    InProceedings of the 40th International Conference on Software Engineering

    When not to comment: Questions and tradeoffs with API documentation for C++ projects. InProceedings of the 40th International Conference on Software Engineering. 643–653

  97. [2021]

    Generating unit tests for documentation.IEEE Transactions on Software Engineering48, 9 (2021), 3268–3279

    2021

  98. [2024]

    In2024 IEEE Symposium on Security and Privacy (SP)

    Poisoned chatgpt finds work for idle hands: Exploring developers’ coding practices with insecure suggestions from poisoned ai models. In2024 IEEE Symposium on Security and Privacy (SP). IEEE, 1141–1159

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.