Pith. sign in

REVIEW 3 major objections 5 minor 89 references

Stack Overflow Meets Replication: Security Research Amid Evolving Code Snippets (Extended Version)

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that security findings drawn from a single Stack Overflow snapshot are time-dependent: four of six replicated studies produced materially different conclusions on a newer dataset version, so the field should treat the…

desk verdict Useful meta-replication with a genuinely new contribution, but the 'four papers changed' headline is partly confounded by tool-version and pipeline changes in the two most striking case studies. read the letter →

arxiv 2501.16948 v3 pith:MEZZ7S6J submitted 2025-01-28 cs.CR

Alfusainey Jallow , Sven Bugiel This is my paper · ORCID
classification cs.CR
keywords StackOverflowcodesnippetevolutionreplicationstudytime-seriesanalysissecurityweaknessesSOTorrentdatasetnon-stationaritycross-sectionalresearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that security research results derived from Stack Overflow are hostage to the moment the data was collected. The authors systematized 42 prior studies, showed that the aspects those studies rely on (programming-language mix, edits to snippets, comments around them) drift over time, and then re-ran six of the studies on newer dataset versions. Four of the six produced conclusions that no longer matched the originals: vulnerability-type rankings shifted, a detection tool's precision fell from 91% to 11%, and the share of insecure Python snippets dropped sharply. Two studies of crypto-API misuse in Java remained stable. The paper's recommendation is that the field treat Stack Overflow as a time-series data source and present findings as trends rather than as timeless cross-sectional facts.

What carries the argument

The central mechanism is paired replication across dataset snapshots: each of six studies is re-run on a newer version of the same underlying data source while keeping the study's method fixed as far as the available artifacts allow. Four of the six relied on SOTorrent, a dataset that tracks individual code-snippet versions; because the dataset's makers stopped publishing releases in 2020, the authors built a fresh release (SOTorrent22) from the June 2022 Stack Exchange dump using the open-source extraction tooling. To show the underlying material is non-stationary, the paper computes the monthly percentage of security-relevant commits (PSC) and applies two stationarity tests (KPSS and ADF). The six replications then serve as the concrete demonstration that conclusions depend on when the data was captured.

What would settle it

Hold the analysis toolchain fixed across data versions, using the same Cppcheck release and the same language-identification method on both the 2018 and 2022 snapshots, and re-run the C/C++ weakness study. The paper's own paired t-test (Appendix B) shows the dataset effect is not consistent across Cppcheck versions, so if a fixed toolchain removes the CWE-ranking and improvement-rate shifts, the drift would be attributed to re-implementation rather than to Stack Overflow's evolution.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that Stack Overflow's code snippets, languages, and security-relevant discussion move continuously, and these movements change what a study concludes. The authors re-ran six prior security studies on newer dataset versions: SOTorrent22, a release they rebuilt from the June 2022 Stack Exchange dump, and the September 2023 Stack Exchange dump. Four of the six produced materially different results: the ranking of CWE types in C/C++ snippets shifted and a new type (CWE-476) appeared; the DICOS tool's precision fell from 91% to 11% and accuracy from 89% to 32%; the share of Python questions with insecure answers dropped from 18.1% to 4.9%; and the claim that every JavaScript snippet violates a rule became false, since nine violation-free snippets exist. The two studies of crypto-API misuse in Java kept their conclusions, which the authors attribute to that niche topic requiring domain expertise, so the evolution is not uniform across topics. The paper concludes that prior results are not wrong but are missing temporal context, and that Stack Overflow research should be reported as trend analysis.

Load-bearing premise

The load-bearing premise is that the replications are faithful enough to the original studies that the observed differences come from Stack Overflow's evolution rather than from the re-implementers' changed tools and version choices.

Editorial extensions

If this is right

  • Stack Overflow-based studies should report their dataset snapshot and version as part of the finding, since the same analysis at a later date can reach a different conclusion.
  • Re-running a prior study on a newer snapshot becomes a meaningful test of the result rather than a formality, and replication studies should be read that way.
  • The measured accuracy of tools built on Stack Overflow data is also time-dependent: the DICOS precision drop from 91% to 11% shows that tool evaluations need a temporal context.
  • Topic niche appears to buffer against drift: the crypto-API misuse studies in Java held stable across seven years, suggesting that findings on specialized, expert-heavy topics may be more durable.
  • The field needs better artifact discipline, such as reporting tool versions and releasing working code and data, otherwise temporal shifts cannot be separated from re-implementation differences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The time-dependence demonstrated for security studies likely extends to non-security Stack Overflow research, such as API-usage or developer-behavior studies, because the underlying data (languages, edits, comments) is the same.
  • A controlled falsification would hold the toolchain fixed across both datasets; the paper's own Appendix B shows the dataset effect varies with the Cppcheck version, so part of the observed shift in the C/C++ study could be tool-driven rather than evolution-driven.
  • One practical standard suggested by the paper's logic: require new Stack Overflow studies to report results on at least two snapshot versions with a pinned toolchain, flagging any claim that flips between them.
  • The rising share of security-relevant comments and the durability of the Java crypto results together hint that community curation concentrates on identifiable high-risk code; a natural next study is whether snippet security improves fastest where such curation is most active.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether the evolution of Stack Overflow content affects the stability of findings from security research that uses Stack Overflow datasets. It contributes (i) a systematic literature review of 42 papers, (ii) a time-series analysis of snippet volume, security-relevant edits, and security-relevant comments, and (iii) six replication case studies on newer dataset versions. The authors report that four of the six replications yielded significantly different results and recommend that researchers treat Stack Overflow as a time-series data source rather than as a static cross-sectional dataset.

Significance. If the replication results are taken at face value, the paper provides a useful cautionary meta-result: cross-sectional Stack Overflow studies may not generalize across dataset versions, and the field should move toward longitudinal designs and open-science practices. The literature review, the stationarity testing with KPSS and ADF, and the public Zenodo artifact are concrete strengths, and the paper is transparent about many of its limitations. However, the central quantitative claim is currently over-identified because two of the four 'significantly different' case studies confound dataset evolution with tool-version and pipeline changes; the paper's own Appendix B shows that the dataset effect is not stable across Cppcheck versions. The qualitative recommendation to treat Stack Overflow as a time series is defensible even after this correction, but the abstract's precise count of 'four papers' needs a stronger evidentiary basis.

major comments (3)
  1. [Section 6.1.2-6.1.3 and Appendix B (Table 23)] The headline comparison for Zhang et al. confounds dataset version with tool-version and candidate-pool construction. The original pipeline used Guesslang for language filtering and Cppcheck v1.86 for weakness detection, while the replication skips Guesslang and uses Cppcheck v2.13 for both tasks. The authors report in Table 23 that switching from Cppcheck v1.86 to v2.13 on the same SOTorrent18 dataset raises the number of weak snippets from 11,748 to 24,699, a change larger than the reported cross-dataset growth (11,748 to 30,254), and the paired t-test (t = -8.90, p < 0.05) shows that the dataset effect is not consistent across Cppcheck versions. Tables 2-4 should therefore be recalculated with a fixed scanner version (e.g., Cppcheck v1.86 on both SOTorrent versions, as the data in Table 23 already permit) before claiming that SOTorrent22 contains proportionally more vulnerable snippets.
  2. [Section 6.2.2-6.2.3 and Appendix C] The DICOS replication does not isolate dataset evolution from implementation drift. The authors could not reproduce the original SQL-based post counts even on SOTorrent20 (867,962 vs 987,367 C/C++ posts; 986,900 vs 970,916 Android posts), and they found 26,550 insecure posts instead of the reported 12,458 before applying bug fixes. Running the bug-fixed tool on SOTorrent22 and comparing the resulting 11% precision to the original 91% therefore attributes the drop to Stack Overflow evolution without a baseline reproduction of the fixed tool on SOTorrent20. Such a baseline is necessary to support the claim that 'code evolution has adversely affected the precision and accuracy of DICOS.'
  3. [Section 7.2] The claim that the two Fischer et al. studies 'remained stable' is not established for the part of the data that the replication cannot see. The labeling analysis only tracks revisions of the original DS2016/DS2018 snippets (with only 9, 3, and 8 security label changes in the three periods), and the authors explicitly state that they cannot classify new snippets added after the original collection. The tag-based proxy shows 24,767 new crypto-related answers after DS2016 (87.6% growth), which is equally consistent with a change in the fraction of vulnerable snippets. The conclusion should be limited to 'the original labeled snippets rarely changed,' not that the overall findings are stable.
minor comments (5)
  1. [Appendix C] The sentence 'accepted (1.67% ↗ 92.83%) and non-accepted (1.99% ↘ 93.7%)' is numerically inconsistent with Section 6.2.3 and Figure 9, where the replicated ratios are 7.72% and 6.61%; the 92.83%/93.7% values appear to be a typo and should be corrected.
  2. [Section 6.3.3] The reported 'p−value (0.9 ↗ 6.2)' is not a valid p-value, as probabilities cannot exceed 1; please clarify whether this is a test statistic or a different quantity.
  3. [Section 3.2] The text says 'Table 1 compares the 43 considered studies' but the table and the rest of the paper report 42 relevant studies; the number should be harmonized.
  4. [Appendix B.2] The discussion of the user who contributed CWE-775 refers to 'CWE-755' in the following sentence; the CWE number should be made consistent.
  5. [Section 6.4.2] In the Campos replication, the statement about 153,159 parse-error snippets in the original data and 267,795 parse-error snippets in SOTorrent22 should explicitly state the dataset version for each number to avoid confusion.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: replication outcomes are measured against external published results; the only self-citation is a disclosed input measurement tool.

full rationale

The six replication case studies compare against externally published tables (e.g., Zhang et al. Table 1 in [87]; Hong et al. Table 5 in [38]) on a newer SOTorrent or StackExchange snapshot. These comparisons are not constructed from the paper's own fitted parameters, so the central claim that four studies change is not true by definition. The one self-citation, 'we apply the publicly shared NLP-based classifier by Jallow et al. [1, 44]' (Section 5), feeds the time-series analysis of security-relevant edits and comments but not any replicated outcome; the authors flag its 'potentially high false positive rate due to its keyword-based detection' (Section 8), and the trend statement would stand or fall on the tool's measurement validity, not on a circular reduction. The known threats to the C/C++ and DICOS replications (skipping Guesslang, choosing Cppcheck v2.13 despite the Appendix B t-test showing dataset effects are not consistent across Cppcheck versions; bug-fixed DICOS and unreproducible original post counts) are validity and confounding concerns, not circularity: the headline comparison is not equivalent to the input by construction. Overall, no load-bearing step reduces to a self-citation or to a fitted input renamed as a prediction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its central claims depend on the fidelity of the new SOTorrent22 dataset, the correctness of external tools (Cppcheck, DICOS, the NLP classifier), the accuracy of the original studies' artifacts, and the hand-chosen tool versions and tag thresholds. The free parameters and assumptions listed above are the main external inputs the conclusions rest on.

free parameters (2)
  • Cppcheck version selection = v1.86 for baseline validation, v2.13 for replication
    The authors brute-forced 15 Cppcheck versions to find the one closest to Zhang et al.'s original results, then used a different version (v2.13) for the newer dataset. This hand-picked choice is essential to the claimed comparison in Section 6.1.
  • Top-50 crypto-related tag threshold = 50
    Two researchers manually selected the top 50 crypto-related co-located tags to build a proxy for new vulnerable snippets in the Fischer et al. case study (Section 7). This threshold is a hand-chosen parameter that affects the estimated lower bound.
assumptions (5)
  • domain assumption SOTorrent22, built with the open-source posthistory-extractor from the June 2022 Stack Exchange dump, faithfully represents the state of Stack Overflow at that time.
    The paper relies on this dataset as the newer snapshot for all replication studies without an independent ground-truth validation of the extractor's correctness.
  • domain assumption Cppcheck can serve as both a C/C++ language detector and a security weakness detector by failing on non-C/C++ snippets.
    This assumption replaces Guesslang in the Zhang et al. replication (Section 6.1.2) and is not independently validated.
  • domain assumption The Jallow et al. NLP classifier correctly labels security-relevant edits and comments despite its known keyword-based false positive rate.
    Used in Section 5 to compute the percentage of security-relevant commits and comments; the authors themselves note the classifier's false positive risk in Section 8.
  • domain assumption The original studies being replicated are accurately described by their published papers and available artifacts.
    Replication conclusions depend on this premise; the paper found several cases where papers and artifacts disagreed, e.g., Campos et al.'s LoC threshold and parse-error handling in Section 6.4.2.
  • standard math Standard stationarity tests (KPSS, ADF) and statistical tests (chi-square, t-tests, z-tests) are appropriate for the time-series and comparison analyses.
    The paper applies these tests without questioning their assumptions, which is standard practice in empirical software engineering.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stack Overflow Meets Replication: Security Research Amid Evolving Code Snippets (Extended Version)." pith.science (2026). https://pith.science/paper/MEZZ7S6J

@misc{pith2026250116948,
  author       = {Pith},
  title        = {Pith review of: Stack Overflow Meets Replication: Security Research Amid Evolving Code Snippets (Extended Version)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MEZZ7S6J}},
  note         = {Machine review of arXiv:2501.16948}
}
read the original abstract

We study the impact of Stack Overflow code evolution on the stability of prior research findings derived from Stack Overflow data and provide recommendations for future studies. We systematically reviewed papers published between 2005--2023 to identify key aspects of Stack Overflow that can affect study results, such as the language or context of code snippets. Our analysis reveals that certain aspects are non-stationary over time, which could lead to different conclusions if experiments are repeated at different times. We replicated six studies using a more recent dataset to demonstrate this risk. Our findings show that four papers produced significantly different results than the original findings, preventing the same conclusions from being drawn with a newer dataset version. Consequently, we recommend treating Stack Overflow as a time series data source to provide context for interpreting cross-sectional research conclusions.

Figures

Figures reproduced from arXiv: 2501.16948 by the authors.

Figure 1
Figure 1. First version of the answer (left-hand side) labeled [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. PRISMA diagram of our literature review 3.1 Literature Search We conducted a systematic literature review following the guidelines by Kitchenham and Charters [47]. Inclusion and Exclusion Criteria. The inclusion criteria are that the study: IC1. must focus on the Stack Overflow website. IC2. must examine code snippets on Stack Overflow. IC3. must analyze the security of code snippets or identify and address bugs or … view at source ↗
Figure 3
Figure 3. Added code snippets on Stack Overflow per month. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (28 more)
Figure 4
Figure 4. Figure 4: Number of monthly (30-day interval) post edits [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Percentage of security-relevant commits (PSC) in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Zhang et al.’s [87] methodology with the approach used in our replication study. to determine the language of code snippets, i.e., not all snip￾pets contain valid C/C++ code. Using the Guesslang machine learning classifier [32] they filtered non-C/C++ cod…
Figure 7
Figure 7. Figure 7: The accumulative proportion of Versionw posted by the proportion of users. Annotations indicate the original (cf [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 9
Figure 9. Figure 9: Ratio of insecure posts between accepted and non [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Types of discovered insecure code snippets with [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Nr. of monthly created posts with Java and/or An￾droid tag plus at least one top-50 crypto-related tag. Vertical lines indicate data collection points by Fischer et al. [26, 28] [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 13
Figure 13. Figure 13: Frequency of co-location between top-50 crypto [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: The evolution of an answer post in DS2016 between March 4 th, 2016 and September 3 rd, 2023. Two additional versions have been created during this time interval. The first version shows the post on March 4 th, 2016, labeled by Fischer et al. as secure (top snippet) an…
Figure 15
Figure 15. Figure 15: Normalized confusion matrix for [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 17
Figure 17. Figure 17: C language: PSC in monthly intervals. Dashed line is the fitted linear regression. 2010 2012 2014 2016 2018 2020 2022 10 1 10 2 10 3 Nr. of post edits / month No commit message Not Security-Relevant Security-Relevant (a) Nr. of monthly post edits categorized by their …
Figure 16
Figure 16. Figure 16: Normalized confusion matrix for [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 18
Figure 18. Figure 18: C++ language: PSC in monthly intervals. Dashed line is the fitted linear regression. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]
Figure 19
Figure 19. Figure 19: Java language: PSC in monthly intervals. Dashed line is the fitted linear regression [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 23
Figure 23. Figure 23: Security-relevant comments on Stack Overflow for [PITH_FULL_IMAGE:figures/full_fig_p031_23.png]
Figure 24
Figure 24. Figure 24: Security-relevant comments on Stack Overflow for [PITH_FULL_IMAGE:figures/full_fig_p031_24.png]
Figure 25
Figure 25. Figure 25: Security-relevant comments on Stack Overflow for [PITH_FULL_IMAGE:figures/full_fig_p032_25.png]
Figure 27
Figure 27. Figure 27: Security-relevant comments on Stack Overflow for [PITH_FULL_IMAGE:figures/full_fig_p032_27.png]
Figure 26
Figure 26. Figure 26: Security-relevant comments on Stack Overflow for [PITH_FULL_IMAGE:figures/full_fig_p032_26.png]
Figure 28
Figure 28. Figure 28: The distribution of user reputation points for users [PITH_FULL_IMAGE:figures/full_fig_p033_28.png]
Figure 29
Figure 29. Figure 29: Number of distinct CWE types introduced by users. [PITH_FULL_IMAGE:figures/full_fig_p033_29.png]
Figure 30
Figure 30. Figure 30: The number/proportion of users contributing a [PITH_FULL_IMAGE:figures/full_fig_p034_30.png]
Figure 32
Figure 32. Figure 32: The distribution of entropy for CWE instances of [PITH_FULL_IMAGE:figures/full_fig_p035_32.png]
Figure 33
Figure 33. Figure 33: The number/proportion of users contributing a [PITH_FULL_IMAGE:figures/full_fig_p035_33.png]
Figure 34
Figure 34. Figure 34: The distribution of contributed CWE instances by [PITH_FULL_IMAGE:figures/full_fig_p036_34.png]
Figure 36
Figure 36. Figure 36: Comparison of yearly distribution of secure and [PITH_FULL_IMAGE:figures/full_fig_p036_36.png]
Figure 35
Figure 35. Figure 35: The distribution of entropy for CWE instances of [PITH_FULL_IMAGE:figures/full_fig_p036_35.png]
Figure 37
Figure 37. Figure 37: Intersection between the manually verified lan [PITH_FULL_IMAGE:figures/full_fig_p038_37.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 78 canonical work pages

  1. [1]

    Artifact for Paper: Measuring the Effects of Stack Overflow Code Snippet Evolution

    “Artifact for Paper: Measuring the Effects of Stack Overflow Code Snippet Evolution.” [Online]. Available: https://osf.io/ s2vgm/?view_only=785ada7b1efd4ac6aaf5a77cc5123076

  2. [2]

    Artifact for Paper: Snakes in Paradies

    “Artifact for Paper: Snakes in Paradies.” [Online]. Available: https://figshare.com/s/588b0d450310c05d25ab? file=14644805

  3. [3]

    SIGPLAN Empirical Evaluation Guidelines

    “SIGPLAN Empirical Evaluation Guidelines.” [Online]. Avail- able: https://www.sigplan.org/Resources/EmpiricalEvaluation/

  4. [4]

    On code reuse from stackoverflow: An exploratory study on android apps,

    R. Abdalkareem, E. Shihab, and J. Rilling, “On code reuse from stackoverflow: An exploratory study on android apps,” Information and Software Technology, vol. 88, pp. 148–158, 2017

    2017

  5. [5]

    You get where you’re looking for: The impact of information sources on code security,

    Y . Acar, M. Backes, S. Fahl, D. Kim, M. L. Mazurek, and C. Stransky, “You get where you’re looking for: The impact of information sources on code security,” inProc. 37th IEEE Symposium on Security and Privacy (SP ’16). IEEE Computer Society, 2016

    2016

  6. [6]

    A decade of Privacy-Relevant android app reviews: Large scale trends,

    O. Akgul, S. T. Peddinti, N. Taft, M. L. Mazurek, H. Harkous, A. Srivastava, and B. Seguin, “A decade of Privacy-Relevant android app reviews: Large scale trends,” in 33rd USENIX Security Symposium (USENIX Security 24). USENIX Asso- ciation, 2024

    2024

  7. [7]

    Towards best secure coding prac- tice for implementing ssl/tls,

    M. Alhanahnah and Q. Yan, “Towards best secure coding prac- tice for implementing ssl/tls,” inIEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFO- COM WKSHPS), 2018

    2018

  8. [8]

    Rexstep- per: a reference debugger for javascript regular expressions,

    L. Almeida, M. Gonzaga, J. F. Santos, and R. Abreu, “Rexstep- per: a reference debugger for javascript regular expressions,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) , 2023

    2023

Show all 89 references
  1. [9]

    Apache Solr Website,

    Apache Solr, “Apache Solr Website,” accessed 2019-12-09. [Online]. Available: https://lucene.apache.org/solr/

    2019

  2. [10]

    Actor concurrency bugs: a com- prehensive study on symptoms, root causes, api usages, and differences,

    M. Bagherzadeh, N. Fireman, A. Shawesh, and R. Khatchadourian, “Actor concurrency bugs: a com- prehensive study on symptoms, root causes, api usages, and differences,” Proc. ACM Program. Lang., vol. 4, no. OOPSLA, Nov. 2020. 7https://meta.stackexchange.com/questions/334811/ st...

    2020

  3. [11]

    A qualitative investiga- tion of insecure code propagation from online forums,

    W. Bai, O. Akgul, and M. L. Mazurek, “A qualitative investiga- tion of insecure code propagation from online forums,” in2019 IEEE Cybersecurity Development (SecDev), 2019, pp. 34–48

    2019

  4. [12]

    Usage and attribution of stack over- flow code snippets in github projects,

    S. Baltes and S. Diehl, “Usage and attribution of stack over- flow code snippets in github projects,” Empirical Software Engineering, vol. 24, no. 3, pp. 1259–1295, Jun 2019

    2019

  5. [13]

    Sotorrent: Studying the origin, evolution, and usage of stack overflow code snippets,

    S. Baltes, C. Treude, and S. Diehl, “Sotorrent: Studying the origin, evolution, and usage of stack overflow code snippets,” in Proc. 16th International Conference on Mining Software Repositories (MSR 2019), 2019

    2019

  6. [14]

    J. M. Box-Steffensmeier, J. R. Freeman, M. P. Hitt, and J. C. W. Pevehouse, Time Series Analysis for the Social Sciences, ser. Analytical Methods for Social Research. Cambridge Univer- sity Press, 2014

    2014

  7. [15]

    Information evolution in wikipedia,

    A. Ceroni, M. Georgescu, U. Gadiraju, K. D. Naini, and M. Fisichella, “Information evolution in wikipedia,” in Pro- ceedings of The International Symposium on Open Collabora- tion, ser. OpenSym ’14. ACM, 2014

    2014

  8. [16]

    Does reusing pre-trained nlp model propa- gate bugs?

    M. Chakraborty, “Does reusing pre-trained nlp model propa- gate bugs?” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 2021

    2021

  9. [17]

    Crowd debugging,

    F. Chen and S. Kim, “Crowd debugging,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engi- neering, ser. ESEC/FSE 2015. ACM, 2015

    2015

  10. [18]

    itrustso: An intelligent system for automatic detection of insecure code snippets in stack overflow,

    L. Chen, S. Hou, Y . Ye, T. Bourlai, S. Xu, and L. Zhao, “itrustso: An intelligent system for automatic detection of insecure code snippets in stack overflow,” in2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Min- ing (ASONAM), 2019

    2019

  11. [19]

    How reliable is the crowdsourced knowledge of security imple- mentation?

    M. Chen, F. Fischer, N. Meng, X. Wang, and J. Grossklags, “How reliable is the crowdsourced knowledge of security imple- mentation?” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), 2019

    2019

  12. [20]

    Dagenais

    B. Dagenais. (2022) Partial program analysis for eclipse. [Online]. Available: https://www.sable.mcgill.ca/ppa/ppa_ eclipse.html

    2022

  13. [21]

    Reproducibility and replicability of web measurement studies,

    N. Demir, M. Große-Kampmann, T. Urban, C. Wressnegger, T. Holz, and N. Pohlmann, “Reproducibility and replicability of web measurement studies,” in Proceedings of the ACM Web Conference 2022, 2022

    2022

  14. [22]

    Systematizing systematization of knowledge

    D. Evans, “Systematizing systematization of knowledge.” [Online]. Available: https://oaklandsok.github.io/

  15. [23]

    Mining rule violations in javascript code snip- pets,

    U. Ferreira Campos, G. Smethurst, J. P. Moraes, R. Bonifácio, and G. Pinto, “Mining rule violations in javascript code snip- pets,” in 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), 2019

    2019

  16. [24]

    A large- scale study of the evolution of web pages,

    D. Fetterly, M. Manasse, M. Najork, and J. Wiener, “A large- scale study of the evolution of web pages,” in Proceedings of the 12th International Conference on World Wide Web, ser. WWW ’03. ACM, 2003

    2003

  17. [25]

    On the use of c# unsafe code context: An empirical study of stack overflow,

    E. Firouzi, A. Sami, F. Khomh, and G. Uddin, “On the use of c# unsafe code context: An empirical study of stack overflow,” in Proc. 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) . ACM, 2020

    2020

  18. [26]

    Stack overflow considered harmful? the impact of copy&paste on android application security,

    F. Fischer, K. Böttinger, H. Xiao, C. Stransky, Y . Acar, M. Backes, and S. Fahl, “Stack overflow considered harmful? the impact of copy&paste on android application security,” in Proc. 38th IEEE Symposium on Security and Privacy (SP ’17). IEEE Computer Society, 2017

    2017

  19. [27]

    The effect of google search on software security: Unobtrusive security interventions via content re-ranking,

    F. Fischer, Y . Stachelscheid, and J. Grossklags, “The effect of google search on software security: Unobtrusive security interventions via content re-ranking,” in Proc. 28th ACM Con- ference on Computer and Communication Security (CCS’21). ACM, 2021

    2021

  20. [28]

    Stack overflow considered helpful! deep learning security nudges towards stronger cryptography,

    F. Fischer, H. Xiao, C.-Y . Kao, Y . Stachelscheid, B. Johnson, D. Razar, P. Fawkesley, N. Buckley, K. Böttinger, P. Muntean, and J. Grossklags, “Stack overflow considered helpful! deep learning security nudges towards stronger cryptography,” in Proc. 28th USENIX Security Symp...

    2019

  21. [29]

    Fuller,Introduction to statistical time series, ser

    W. Fuller,Introduction to statistical time series, ser. A Wiley publication in applied statistics. Wiley, 1976

    1976

  22. [30]

    Fixing recurring crash bugs via analyzing q&a sites (t),

    Q. Gao, H. Zhang, J. Wang, Y . Xiong, L. Zhang, and H. Mei, “Fixing recurring crash bugs via analyzing q&a sites (t),” in Proc. 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2015

    2015

  23. [31]

    Mutation-based fault localization of deep neural networks,

    A. Ghanbari, D.-G. Thomas, M. A. Arshad, and H. Rajan, “Mutation-based fault localization of deep neural networks,” in Proc. 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE Press, 2024

    2024

  24. [32]

    (2021) Guesslang documentation

    Guesslang. (2021) Guesslang documentation. [Online]. Available: https://guesslang.readthedocs.io/en/latest/

    2021

  25. [33]

    J. D. Hamilton, Time Series Analysis. Princeton University Press, 1994

    1994

  26. [34]

    Hark: A deep learning system for navigating privacy feedback at scale,

    H. Harkous, S. T. Peddinti, R. Khandelwal, A. Srivastava, and N. Taft, “Hark: A deep learning system for navigating privacy feedback at scale,” in 43rd IEEE Symposium on Security and Privacy (SP’22). IEEE, 2022

    2022

  27. [35]

    A simple sequentially rejective multiple test pro- cedure,

    S. Holm, “A simple sequentially rejective multiple test pro- cedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979

    1979

  28. [36]

    H. Hong. (2022) Data collection sql script. [Online]. Available: https://github.com/hyunji-Hong/Dicos-public/blob/ main/src/sql/collecting_allhisotyPost.sql

    2022

  29. [37]

    (2022) Dicos github repository

    ——. (2022) Dicos github repository. [Online]. Available: https://github.com/hyunji-Hong/Dicos-public

    2022

  30. [38]

    Dicos: Discovering insecure code snippets from stack overflow posts by leveraging user discussions,

    H. Hong, S. Woo, and H. Lee, “Dicos: Discovering insecure code snippets from stack overflow posts by leveraging user discussions,” in Annual Computer Security Applications Con- ference (ACSAC). ACM, 2021

    2021

  31. [39]

    Time series analysis of copy-and- paste impact on android application security,

    H. Imai and A. Kanaoka, “Time series analysis of copy-and- paste impact on android application security,” in 13th Asia Joint Conference on Information Security (AsiaJCIS), 2018

    2018

  32. [40]

    S. E. Inc. (2024) Data explorer. [Online]. Avail- able: https://data.stackexchange.com/stackoverflow/query/ 1858561/total-number-of-question-and-answer-posts

    2024

  33. [41]

    Why most published research findings are false,

    J. P. A. Ioannidis, “Why most published research findings are false,” PLOS Medicine, vol. 2, no. 8, p. null, 08 2005. 22

    2005

  34. [42]

    Meta-research: Evaluation and improvement of research meth- ods and practices,

    J. P. A. Ioannidis, D. Fanelli, D. D. Dunne, and S. N. Goodman, “Meta-research: Evaluation and improvement of research meth- ods and practices,” PLOS Biology, vol. 13, no. 10, pp. 1–7, 10 2015

    2015

  35. [43]

    Repairing deep neural networks: fix patterns and challenges,

    M. J. Islam, R. Pan, G. Nguyen, and H. Rajan, “Repairing deep neural networks: fix patterns and challenges,” in Proc. ACM/IEEE 42nd International Conference on Software Engi- neering (ICSE). ACM, 2020

    2020

  36. [44]

    Measuring the effects of stack overflow code snippet evolution on open- source software security,

    A. Jallow, M. Schilling, M. Backes, and S. Bugiel, “Measuring the effects of stack overflow code snippet evolution on open- source software security,” in45th IEEE Symposium on Security and Privacy (SP’24). IEEE, 2024

    2024

  37. [45]

    Dataset and code for: Stack overflow meets replication: Security research amid evolving code snippets,

    A. Jallow, “Dataset and code for: Stack overflow meets replication: Security research amid evolving code snippets,” 2025, artifact for the research study on the impact of Stack Overflow code evolution research results. [Online]. Available: https://doi.org/10.5281/zenodo.14733195

    2025 doi

  38. [46]

    A static analyzer for detecting tensor shape errors in deep neural network training code,

    H. Y . Jhoo, S. Kim, W. Song, K. Park, D. Lee, and K. Yi, “A static analyzer for detecting tensor shape errors in deep neural network training code,” inProc. ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (ICSE). ACM, 2022

    2022

  39. [47]

    Guidelines for performing systematic literature reviews in software engineering,

    B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering,” vol. 2, 01 2007

    2007

  40. [48]

    Krippendorff, Content Analysis: An Introduction to Its Methodology (second edition)

    K. Krippendorff, Content Analysis: An Introduction to Its Methodology (second edition). Sage Publications, 2004

    2004

  41. [49]

    Testing the null hypothesis of stationarity against the alterna- tive of a unit root: How sure are we that economic time series have a unit root?

    D. Kwiatkowski, P. C. B. Phillips, P. Schmidt, and Y . Shin, “Testing the null hypothesis of stationarity against the alterna- tive of a unit root: How sure are we that economic time series have a unit root?” Journal of Econometrics, vol. 54, no. 1-3, pp. 159–178, 1992

    1992

  42. [50]

    Contextual profiling of stack overflow java code security vulnerabilities initial insights from a pilot study,

    S. A. Licorish and T. Nishatharan, “Contextual profiling of stack overflow java code security vulnerabilities initial insights from a pilot study,” in21st International Conference on Soft- ware Quality, Reliability and Security Companion (QRS-C) . IEEE Computer Society, 2021

    2021

  43. [51]

    Dissecting copy/delete/re- place/swap mutations: insights from a gin case study,

    S. A. Licorish and M. Wagner, “Dissecting copy/delete/re- place/swap mutations: insights from a gin case study,” inPro- ceedings of the Genetic and Evolutionary Computation Con- ference Companion, ser. GECCO ’22. ACM, 2022

    2022

  44. [52]

    Deep- anna: Deep learning based java annotation recommendation and misuse detection,

    Y . Liu, Y . Yan, C. Sha, X. Peng, B. Chen, and C. Wang, “Deep- anna: Deep learning based java annotation recommendation and misuse detection,” in IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , 2022

    2022

  45. [53]

    Ljung, System identification: theory for the user

    L. Ljung, System identification: theory for the user. Prentice- Hall, Inc., 1986

    1986

  46. [54]

    A model for reasoning about javascript promises,

    M. Madsen, O. Lhoták, and F. Tip, “A model for reasoning about javascript promises,” Proc. ACM Program. Lang. , vol. 1, no. OOPSLA, Oct. 2017. [Online]. Available: https://doi.org/10.1145/3133910

    2017 doi

  47. [55]

    Recommend- ing stack overflow posts for fixing runtime exceptions using failure scenario matching,

    S. Mahajan, N. Abolhassani, and M. R. Prasad, “Recommend- ing stack overflow posts for fixing runtime exceptions using failure scenario matching,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw...

    2020

  48. [56]

    Providing real-time assistance for repairing runtime exceptions using stack overflow posts,

    S. Mahajan and M. R. Prasad, “Providing real-time assistance for repairing runtime exceptions using stack overflow posts,” in IEEE Conference on Software Testing, Verification and Vali- dation (ICST), 2022

    2022

  49. [57]

    Studying the change histories of stack overflow and github snippets,

    S. S. Manes and O. Baysal, “Studying the change histories of stack overflow and github snippets,” in IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), 2021

    2021

  50. [58]

    Secure coding practices in java: Challenges and vulnerabili- ties,

    N. Meng, S. Nagy, D. Yao, W. Zhuang, and G. Arango-Argoty, “Secure coding practices in java: Challenges and vulnerabili- ties,” in IEEE/ACM 40th International Conference on Software Engineering (ICSE), 2018

    2018

  51. [59]

    Fluid intelligence doesn’t mat- ter! effects of code examples on the usability of crypto apis,

    K. Mindermann and S. Wagner, “Fluid intelligence doesn’t mat- ter! effects of code examples on the usability of crypto apis,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings, ser. ICSE ’20. ACM, 2020

    2020

  52. [60]

    µakka: Mutation testing for actor concurrency in akka using real-world bugs,

    M. Moradi Moghadam, M. Bagherzadeh, R. Khatchadourian, and H. Bagheri, “µakka: Mutation testing for actor concurrency in akka using real-world bugs,” in Proc. 31st ACM Joint Euro- pean Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 2023

    2023

  53. [61]

    Does fixing bug increase robustness in deep learning?

    R. Pan, “Does fixing bug increase robustness in deep learning?” in Proc. ACM/IEEE 42nd International Conference on Soft- ware Engineering: Companion Proceedings (ICSE). ACM, 2020

    2020

  54. [62]

    Toxic code snippets on stack overflow,

    C. Ragkhitwetsagul, J. Krinke, M. Paixao, G. Bianco, and R. Oliveto, “Toxic code snippets on stack overflow,” IEEE Transactions on Software Engineering, vol. 47, no. 3, pp. 560– 581, 2021

    2021

  55. [63]

    Snakes in paradise? insecure python-related coding practices in stack overflow,

    A. Rahman, E. Farhana, and N. Imtiaz, “Snakes in paradise? insecure python-related coding practices in stack overflow,” in Proceedings of the 16th International Conference on Mining Software Repositories, ser. MSR ’19. IEEE Press, 2019

    2019

  56. [64]

    An insight into the reusability of stack overflow code fragments in mobile applications,

    M. S. Rahman and C. K. Roy, “An insight into the reusability of stack overflow code fragments in mobile applications,” in IEEE 16th International Workshop on Software Clones (IWSC), 2022

    2022

  57. [65]

    Augmenting stack overflow with api usage patterns mined from github,

    A. Reinhardt, T. Zhang, M. Mathur, and M. Kim, “Augmenting stack overflow with api usage patterns mined from github,” in Proc. 26th ACM Joint Meeting on European Software Engi- neering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2018

    2018

  58. [66]

    Demystify official api usage directives with crowdsourced api misuse scenarios, erroneous code examples and patches,

    X. Ren, J. Sun, Z. Xing, X. Xia, and J. Sun, “Demystify official api usage directives with crowdsourced api misuse scenarios, erroneous code examples and patches,” in IEEE/ACM 42nd International Conference on Software Engineering (ICSE) , 2020

    2020

  59. [67]

    Copypastavulguard – a browser extension to prevent copy and paste spreading of vulnerable source code in forum posts,

    H. Schmidt, M. van Aerssen, C. Leich, A. Benni, S. Al Ali, and J. Tanz, “Copypastavulguard – a browser extension to prevent copy and paste spreading of vulnerable source code in forum posts,” in Proc. 17th International Conference on Availability, Reliability and Security (ARE...

    2022

  60. [68]

    (2022) Sotorrent post history ex- tractor

    Sebastian Baltes. (2022) Sotorrent post history ex- tractor. [Online]. Available: https://github.com/sotorrent/ posthistory-extractor

    2022

  61. [69]

    Does collaborative editing help mitigate security vulnerabilities in crowd-shared iot code ex- amples?

    M. Selvaraj and G. Uddin, “Does collaborative editing help mitigate security vulnerabilities in crowd-shared iot code ex- amples?” in Proc. 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’22). ACM, 2022

    2022

  62. [70]

    Shumway and D

    R. Shumway and D. Stoffer, Time Series Analysis and Its Ap- plications With R Examples, 2011, vol. 9

    2011

  63. [71]

    Analyzing comment-induced updates on stack overflow,

    A. Soni and S. Nadi, “Analyzing comment-induced updates on stack overflow,” inProc. 16th International Conference on Mining Software Repositories (MSR ’19). IEEE Press, 2019

    2019

  64. [72]

    source tool

    O. source tool. (2022) Joern: The bug hunter’s workbench. [Online]. Available: https://joern.io/

    2022

  65. [73]

    Stack Exchange Data Dump),

    I. Stack Exchange, “Stack Exchange Data Dump),” accessed 2023-06-13. [Online]. Available: https://archive.org/details/ stackexchange/

    2023

  66. [74]

    (2022) Academic pa- pers using stack exchange data

    Stack Exchange Meta. (2022) Academic pa- pers using stack exchange data. [Online]. Avail- able: https://meta.stackexchange.com/questions/134495/ academic-papers-using-stack-exchange-data

    2022

  67. [75]

    (2024) Revisions of answer post 14424800

    Stack Overflow. (2024) Revisions of answer post 14424800. [Online]. Available: https://stackoverflow.com/ posts/14424800/revisions

    2024

  68. [76]

    Subramanian

    S. Subramanian. (2022) Java snippet parser. [Online]. Available: https://github.com/siddhukrs/java-baker

    2022

  69. [77]

    Snapshot metrics are not enough: Analyzing soft- ware repositories with longitudinal metrics,

    N. M. Synovic, M. Hyatt, R. Sethi, S. Thota, Shilpika, A. J. Miller, W. Jiang, E. S. Amobi, A. Pinderski, K. Läufer, N. J. Hayward, N. Klingensmith, J. C. Davis, and G. K. Thiru- vathukal, “Snapshot metrics are not enough: Analyzing soft- ware repositories with longitudinal me...

    2023

  70. [78]

    Wikipevent: Leveraging wikipedia edit his- tory for event detection,

    T. Tran, A. Ceroni, M. Georgescu, K. Djafari Naini, and M. Fisichella, “Wikipevent: Leveraging wikipedia edit his- tory for event detection,” in International Conference on Web Information Systems Engineering. Springer, 2014

    2014

  71. [79]

    Tsay, Analysis of financial time series, 2nd ed., ser

    R. Tsay, Analysis of financial time series, 2nd ed., ser. Wiley series in probability and statistics. Wiley-Interscience, 2005

    2005

  72. [80]

    Sok: Benchmarking flaws in systems security,

    E. van der Kouwe, G. Heiser, D. Andriesse, H. Bos, and C. Giuf- frida, “Sok: Benchmarking flaws in systems security,” in Euro- pean Symposium on Security and Privacy (EuroSP). IEEE Computer Society, 2019

    2019

  73. [81]

    An empirical study of c++ vulnerabilities in crowd-sourced code examples,

    M. Verdi, A. Sami, J. Akhondali, F. Khomh, G. Uddin, and A. K. Motlagh, “An empirical study of c++ vulnerabilities in crowd-sourced code examples,” arXiv:1910.01321, 2019

    1910 arXiv

  74. [82]

    We’re switching to commonmark,

    H. V ocke, “We’re switching to commonmark,” https://meta.stackexchange.com/questions/348746/ were-switching-to-commonmark, 2020

    2020

  75. [83]

    Essential guidelines for computational method benchmarking,

    L. M. Weber, W. Saelens, R. Cannoodt, C. Soneson, A. Hapfelmeier, P. P. Gardner, A.-L. Boulesteix, Y . Saeys, and M. D. Robinson, “Essential guidelines for computational method benchmarking,” Genome biology, vol. 20, pp. 1–12, 2019

    2019

  76. [84]

    D. S. Wilks, Statistical methods in the atmospheric sciences. Elsevier Academic Press, 2011

    2011

  77. [85]

    (partial) program dependence learning,

    A. Yadavally, T. N. Nguyen, W. Wang, and S. Wang, “(partial) program dependence learning,” in 2023 IEEE/ACM 45th Inter- national Conference on Software Engineering (ICSE), 2023

    2023

  78. [86]

    Icsd: An automatic system for insecure code snip- pet detection in stack overflow over heterogeneous information network,

    Y . Ye, S. Hou, L. Chen, X. Li, L. Zhao, S. Xu, J. Wang, and Q. Xiong, “Icsd: An automatic system for insecure code snip- pet detection in stack overflow over heterogeneous information network,” in Proceedings of the 34th Annual Computer Secu- rity Applications Conference, ser...

    2018

  79. [87]

    A study of c/c++ code weaknesses on stack overflow,

    H. Zhang, S. Wang, H. Li, T. Chen, and A. E. Hassan, “A study of c/c++ code weaknesses on stack overflow,”IEEE Transac- tions on Software Engineering, vol. 48, no. 07, pp. 2359–2375, jul 2022

    2022

  80. [88]

    Are code examples on an online q&a forum reliable?: A study of api misuse on stack overflow,

    T. Zhang, G. Upadhyaya, A. Reinhardt, H. Rajan, and M. Kim, “Are code examples on an online q&a forum reliable?: A study of api misuse on stack overflow,” inProc. 40th International Conference on Software Engineering (ICSE’18), 2018

    2018

  81. [89]

    An empirical study on tensorflow program bugs,

    Y . Zhang, Y . Chen, S.-C. Cheung, Y . Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in Proc. 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, 2018. 24 A Systematization and Evolution A.1 OpenAI GPT4o Evaluation To...

    2018

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.