{"id":"4132670e-3f52-47bb-9eb2-f4aa3e1627c9","arxiv_id":"2501.16948","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Replicating six Stack Overflow security studies on a newer dataset changed the conclusions of four, showing that cross-sectional findings on evolving code lack temporal context.","lead":"This paper shows that Stack Overflow code and comments evolve over time, and re-running six earlier security studies on a newer snapshot changed the results of four of them. It argues that researchers should treat Stack Overflow data as a time series rather than as a fixed snapshot.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'significantly different' claim confounds dataset evolution with tool-version and pipeline changes in the C/C++ and DICOS replications; a fixed-tool comparison is needed.","rationale":"The reader's weakest_assumption already identifies the fidelity of the replication pipelines as the load-bearing issue, and the C/C++ case is the clearest instance: the original Guesslang step was dropped, Cppcheck was used for both language filtering and weakness detection, and v2.13 was hand-selected despite the authors' own t-test showing that dataset effects are not stable across Cppcheck versions. This is not merely a stylistic mismatch; it directly affects the numbers used to support the 'four papers produced significantly different results' claim. The DICOS case compounds the concern because the original post counts could not be reproduced and a bug-fixed tool was used. None of this attacks the authors' honesty: they report the confounding evidence transparently in Appendix B, which is a credit to the paper. However, the abstract and Section 10 state the conclusion more strongly than the cleanly controlled evidence warrants. Other replications are less affected: Rahman et al. and Campos et al. used published artifacts, reproduced original counts on the old snapshot, and then applied the same pipeline to a newer snapshot, so those cases support the general time-dependence thesis. The reader's CONDITIONAL verdict already captures the right balance: the direction of the finding is credible, but the quantitative headline should be scoped to account for the tool-version and pipeline confounds. My stress-test therefore does not move the verdict; it reinforces the need for a fixed-tool comparison before the strongest quantitative claim is taken at face value.","tokens_in":47155,"tokens_out":4054,"duration_ms":35704,"concrete_test":"Run the exact replication pipeline (Cppcheck v2.13, no Guesslang) on SOTorrent18 and compare the RQ1/RQ2 metrics in Tables 2-4 with the same pipeline on SOTorrent22; then repeat the comparison using the original-style pipeline (Guesslang plus Cppcheck v1.86) on both snapshots. If the CWE rank shifts, the drop in improved-Codew proportions (30.1% to 3.1%), and the entropy changes persist under fixed tool versions, the evolution interpretation survives. If they disappear or reverse, the headline claim must be re-scoped to 'changed pipeline plus changed data' rather than Stack Overflow evolution alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the four 'significantly different' outcomes isolate Stack Overflow evolution from methodological drift. Case Study 1 (Section 6.1.2) violates this premise: the original Zhang et al. pipeline used Guesslang for language filtering plus Cppcheck v1.86 for weakness detection, while the replication skips Guesslang and uses Cppcheck v2.13 for both tasks. The authors themselves report (Appendix B, Table 23, paired t-test t = -8.90, p < 0.05) that the dataset effect is not consistent across Cppcheck versions. Thus the headline comparison in Tables 2-4 confounds dataset version with scanner version and candidate-pool construction: switching to v2.13 alone raises detected snippets on SOTorrent18 from 11,748 to 24,699 (Table 23), a change larger than the reported cross-dataset growth. The DICOS replication (Section 6.2) has a related problem: original post counts could not be reproduced even with the original SQL queries, and the tool was bug-fixed before being applied to the newer snapshot. A reader therefore cannot currently tell how much of the claimed shift is due to the newer SOTorrent version and how much to the altered pipeline. This does not refute the paper's qualitative recommendation to treat Stack Overflow as a time series, but it does undermine the precise abstract claim that 'four papers produced significantly different results' attributable to Stack Overflow evolution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether the evolution of Stack Overflow content affects the stability of findings from security research that uses Stack Overflow datasets. It contributes (i) a systematic literature review of 42 papers, (ii) a time-series analysis of snippet volume, security-relevant edits, and security-relevant comments, and (iii) six replication case studies on newer dataset versions. The authors report that four of the six replications yielded significantly different results and recommend that researchers treat Stack Overflow as a time-series data source rather than as a static cross-sectional dataset.","tokens_in":47341,"tokens_out":6174,"duration_ms":46865,"significance":"If the replication results are taken at face value, the paper provides a useful cautionary meta-result: cross-sectional Stack Overflow studies may not generalize across dataset versions, and the field should move toward longitudinal designs and open-science practices. The literature review, the stationarity testing with KPSS and ADF, and the public Zenodo artifact are concrete strengths, and the paper is transparent about many of its limitations. However, the central quantitative claim is currently over-identified because two of the four 'significantly different' case studies confound dataset evolution with tool-version and pipeline changes; the paper's own Appendix B shows that the dataset effect is not stable across Cppcheck versions. The qualitative recommendation to treat Stack Overflow as a time series is defensible even after this correction, but the abstract's precise count of 'four papers' needs a stronger evidentiary basis.","major_comments":[{"comment":"The headline comparison for Zhang et al. confounds dataset version with tool-version and candidate-pool construction. The original pipeline used Guesslang for language filtering and Cppcheck v1.86 for weakness detection, while the replication skips Guesslang and uses Cppcheck v2.13 for both tasks. The authors report in Table 23 that switching from Cppcheck v1.86 to v2.13 on the same SOTorrent18 dataset raises the number of weak snippets from 11,748 to 24,699, a change larger than the reported cross-dataset growth (11,748 to 30,254), and the paired t-test (t = -8.90, p < 0.05) shows that the dataset effect is not consistent across Cppcheck versions. Tables 2-4 should therefore be recalculated with a fixed scanner version (e.g., Cppcheck v1.86 on both SOTorrent versions, as the data in Table 23 already permit) before claiming that SOTorrent22 contains proportionally more vulnerable snippets.","section":"Section 6.1.2-6.1.3 and Appendix B (Table 23)"},{"comment":"The DICOS replication does not isolate dataset evolution from implementation drift. The authors could not reproduce the original SQL-based post counts even on SOTorrent20 (867,962 vs 987,367 C/C++ posts; 986,900 vs 970,916 Android posts), and they found 26,550 insecure posts instead of the reported 12,458 before applying bug fixes. Running the bug-fixed tool on SOTorrent22 and comparing the resulting 11% precision to the original 91% therefore attributes the drop to Stack Overflow evolution without a baseline reproduction of the fixed tool on SOTorrent20. Such a baseline is necessary to support the claim that 'code evolution has adversely affected the precision and accuracy of DICOS.'","section":"Section 6.2.2-6.2.3 and Appendix C"},{"comment":"The claim that the two Fischer et al. studies 'remained stable' is not established for the part of the data that the replication cannot see. The labeling analysis only tracks revisions of the original DS2016/DS2018 snippets (with only 9, 3, and 8 security label changes in the three periods), and the authors explicitly state that they cannot classify new snippets added after the original collection. The tag-based proxy shows 24,767 new crypto-related answers after DS2016 (87.6% growth), which is equally consistent with a change in the fraction of vulnerable snippets. The conclusion should be limited to 'the original labeled snippets rarely changed,' not that the overall findings are stable.","section":"Section 7.2"}],"minor_comments":[{"comment":"The sentence 'accepted (1.67% ↗ 92.83%) and non-accepted (1.99% ↘ 93.7%)' is numerically inconsistent with Section 6.2.3 and Figure 9, where the replicated ratios are 7.72% and 6.61%; the 92.83%/93.7% values appear to be a typo and should be corrected.","section":"Appendix C"},{"comment":"The reported 'p−value (0.9 ↗ 6.2)' is not a valid p-value, as probabilities cannot exceed 1; please clarify whether this is a test statistic or a different quantity.","section":"Section 6.3.3"},{"comment":"The text says 'Table 1 compares the 43 considered studies' but the table and the rest of the paper report 42 relevant studies; the number should be harmonized.","section":"Section 3.2"},{"comment":"The discussion of the user who contributed CWE-775 refers to 'CWE-755' in the following sentence; the CWE number should be made consistent.","section":"Appendix B.2"},{"comment":"In the Campos replication, the statement about 153,159 parse-error snippets in the original data and 267,795 parse-error snippets in SOTorrent22 should explicitly state the dataset version for each number to avoid confusion.","section":"Section 6.4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its own confounds, which is commendable, but the abstract and main-text framing currently overstate the evidence. A revision that adds the fixed-tool Cppcheck comparison and the DICOS baseline would make the four-paper count credible. The paper fits the journal's scope and would be a useful contribution if the replication claims are tightened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, it does something nobody has done before: it takes six published Stack Overflow security studies, re-runs their pipelines on a newer SOTorrent build it creates itself, and shows that four of them would reach different conclusions. That is a real and timely contribution, and the methodological appendices are honest enough to be useful. Second, the headline claim \"four papers produced significantly different results\" is stronger than the evidence supports, because in the two most striking cases the dataset change is tangled up with pipeline changes. The overall direction is probably right; the specific count is not fully earned.\n\nThe best parts: the 42-paper systematization with the D1–D5/R1–R4 taxonomy is careful work, and the time-series analysis of security-relevant edits and comments adds independent evidence that the platform is non-stationary. The Zhang et al. replication's Appendix B deserves credit for reporting the Cppcheck version instability (paired t-test t=-8.90, p<0.05) and then explaining why they still chose v2.13. That is transparent and the rationale is arguable. The Fischer et al. pair is handled as well as it can be without the original classifier: they track the labeled snippets over seven years, find only nine security-label changes, and use a tag proxy for new posts with explicit caveats. Shipped artifact on Zenodo with reproduceable data collection — that is all in good shape.\n\nThe soft spots are real but narrow. Case Study 1 confounds dataset version with scanner version and candidate-pool construction: skipping Guesslang and switching to Cppcheck v2.13 alone moves the SOTorrent18 detection from 11,748 to 24,699 snippets, a change larger than the reported cross-dataset growth. That is not a minor detail; it is the load-bearing comparison. The DICOS replication has a related problem: original post counts could not be reproduced even with the original SQL, and the tool was bug-fixed before running on the new snapshot. The precision drop from 91% to 11% is then not cleanly attributable to code evolution. The authors flag most of this in the text, which mitigates the damage, but the abstract does not carry those caveats. The \"four papers\" wording should really be \"differences consistent with dataset evolution but partly confounded by methodology changes.\"\n\nBottom line: this deserves a serious referee and, after revision, acceptance. It is exactly the kind of meta-research the field needs, and the recommendation to treat Stack Overflow as time-series data is well argued. I would push the authors to either add a fixed-tool comparison (Cppcheck v1.86 on both datasets, for instance) or rephrase the central claim to separate what is measured from what is inferred.","headline":"Useful meta-replication with a genuinely new contribution, but the 'four papers changed' headline is partly confounded by tool-version and pipeline changes in the two most striking case studies.","tokens_in":47935,"tokens_out":1869,"would_cite":true,"duration_ms":18014,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that security findings drawn from a single Stack Overflow snapshot are time-dependent: four of six replicated studies produced materially different conclusions on a newer dataset version, so the field should treat the…","keywords":["Stack Overflow","code snippet evolution","replication study","time-series analysis","security weaknesses","SOTorrent dataset","non-stationarity","cross-sectional research"],"falsifier":"Hold the analysis toolchain fixed across data versions, using the same Cppcheck release and the same language-identification method on both the 2018 and 2022 snapshots, and re-run the C/C++ weakness study. The paper's own paired t-test (Appendix B) shows the dataset effect is not consistent across Cppcheck versions, so if a fixed toolchain removes the CWE-ranking and improvement-rate shifts, the drift would be attributed to re-implementation rather than to Stack Overflow's evolution.","tokens_in":46863,"feed_emoji":"⏳","tokens_out":11182,"duration_ms":89897,"temperature":0.7,"pith_summary":"This paper argues that security research results derived from Stack Overflow are hostage to the moment the data was collected. The authors systematized 42 prior studies, showed that the aspects those studies rely on (programming-language mix, edits to snippets, comments around them) drift over time, and then re-ran six of the studies on newer dataset versions. Four of the six produced conclusions that no longer matched the originals: vulnerability-type rankings shifted, a detection tool's precision fell from 91% to 11%, and the share of insecure Python snippets dropped sharply. Two studies of crypto-API misuse in Java remained stable. The paper's recommendation is that the field treat Stack Overflow as a time-series data source and present findings as trends rather than as timeless cross-sectional facts.","feed_headline":"Four of six security studies flip on newer Stack Overflow data","feed_subtitle":"Same studies, newer snapshot: vulnerability rankings shift and one detector's precision drops from 91% to 11%.","key_machinery":"The central mechanism is paired replication across dataset snapshots: each of six studies is re-run on a newer version of the same underlying data source while keeping the study's method fixed as far as the available artifacts allow. Four of the six relied on SOTorrent, a dataset that tracks individual code-snippet versions; because the dataset's makers stopped publishing releases in 2020, the authors built a fresh release (SOTorrent22) from the June 2022 Stack Exchange dump using the open-source extraction tooling. To show the underlying material is non-stationary, the paper computes the monthly percentage of security-relevant commits (PSC) and applies two stationarity tests (KPSS and ADF). The six replications then serve as the concrete demonstration that conclusions depend on when the data was captured.","core_discovery":"On the paper's own terms, the discovery is that Stack Overflow's code snippets, languages, and security-relevant discussion move continuously, and these movements change what a study concludes. The authors re-ran six prior security studies on newer dataset versions: SOTorrent22, a release they rebuilt from the June 2022 Stack Exchange dump, and the September 2023 Stack Exchange dump. Four of the six produced materially different results: the ranking of CWE types in C/C++ snippets shifted and a new type (CWE-476) appeared; the DICOS tool's precision fell from 91% to 11% and accuracy from 89% to 32%; the share of Python questions with insecure answers dropped from 18.1% to 4.9%; and the claim that every JavaScript snippet violates a rule became false, since nine violation-free snippets exist. The two studies of crypto-API misuse in Java kept their conclusions, which the authors attribute to that niche topic requiring domain expertise, so the evolution is not uniform across topics. The paper concludes that prior results are not wrong but are missing temporal context, and that Stack Overflow research should be reported as trend analysis.","pith_inferences":["The time-dependence demonstrated for security studies likely extends to non-security Stack Overflow research, such as API-usage or developer-behavior studies, because the underlying data (languages, edits, comments) is the same.","A controlled falsification would hold the toolchain fixed across both datasets; the paper's own Appendix B shows the dataset effect varies with the Cppcheck version, so part of the observed shift in the C/C++ study could be tool-driven rather than evolution-driven.","One practical standard suggested by the paper's logic: require new Stack Overflow studies to report results on at least two snapshot versions with a pinned toolchain, flagging any claim that flips between them.","The rising share of security-relevant comments and the durability of the Java crypto results together hint that community curation concentrates on identifiable high-risk code; a natural next study is whether snippet security improves fastest where such curation is most active."],"forward_implications":["Stack Overflow-based studies should report their dataset snapshot and version as part of the finding, since the same analysis at a later date can reach a different conclusion.","Re-running a prior study on a newer snapshot becomes a meaningful test of the result rather than a formality, and replication studies should be read that way.","The measured accuracy of tools built on Stack Overflow data is also time-dependent: the DICOS precision drop from 91% to 11% shows that tool evaluations need a temporal context.","Topic niche appears to buffer against drift: the crypto-API misuse studies in Java held stable across seven years, suggesting that findings on specialized, expert-heavy topics may be more durable.","The field needs better artifact discipline, such as reporting tool versions and releasing working code and data, otherwise temporal shifts cannot be separated from re-implementation differences."],"supporting_citations":[{"why":"The SOTorrent snippet-version dataset that four of the six replicated studies used; the replications required a newer release of it.","marker":"[13]"},{"why":"The JavaScript ESLint study (case study 4) whose claim that no snippet is violation-free no longer holds on SOTorrent22.","marker":"[23]"},{"why":"The 2016 Java crypto-API study whose labeled snippets the authors tracked over seven years, providing the stable comparison case.","marker":"[26]"},{"why":"The 2018 follow-up Java crypto-API dataset whose labels also remained mostly stable, supporting the niche-topic explanation.","marker":"[28]"},{"why":"The DICOS insecure-snippet detection study (case study 2) whose precision and accuracy collapse when re-run on SOTorrent22.","marker":"[38]"},{"why":"Supplies the NLP classifier used to label post edits as security-relevant for the percentage-of-security-commits (PSC) time series.","marker":"[44]"},{"why":"The Python insecure-practice study (case study 3) whose insecure-answer percentages drop sharply on newer data.","marker":"[63]"},{"why":"The open-source post-history extractor used to build the SOTorrent22 release once the dataset's authors stopped publishing updates.","marker":"[68]"},{"why":"The C/C++ weakness study (case study 1) whose CWE-type ranking and revision-improvement conclusions shift on SOTorrent22.","marker":"[87]"}],"fun_headline_variants":["Four of six security studies flip on newer Stack Overflow data","Stack Overflow evolution flips 4 of 6 study conclusions","Security findings shift as Stack Overflow code evolves","Replicating security studies on fresh data flips four","Stack Overflow data is a moving target for research"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the replications are faithful enough to the original studies that the observed differences come from Stack Overflow's evolution rather than from the re-implementers' changed tools and version choices.","fun_headline_variants_meta":{"raw":{"variants":["Four of six security studies flip on newer Stack Overflow data","Stack Overflow evolution flips 4 of 6 study conclusions","Security findings shift as Stack Overflow code evolves","Replicating security studies on fresh data flips four","Stack Overflow data is a moving target for research"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1250,"prompt_tokens":915,"completion_tokens":335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":258}},"tokens_in":531,"tokens_out":335,"duration_ms":3348,"temperature":1.0,"reasoning_tokens":258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T05:30:12.395046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold the analysis toolchain fixed across data versions, using the same Cppcheck release and the same language-identification method on both the 2018 and 2022 snapshots, and re-run the C/C++ weakness study. The paper's own paired t-test (Appendix B) shows the dataset effect is not consistent across Cppcheck versions, so if a fixed toolchain removes the CWE-ranking and improvement-rate shifts, the drift would be attributed to re-implementation rather than to Stack Overflow's evolution.","supporting_citations":[{"cited_title":"Sotorrent: Studying the origin, evolution, and usage of stack overflow code snippets,","cited_arxiv_id":null,"evidence_quote":"The SOTorrent snippet-version dataset that four of the six replicated studies used; the replications required a newer release of it."},{"cited_title":"Mining rule violations in javascript code snip- pets,","cited_arxiv_id":null,"evidence_quote":"The JavaScript ESLint study (case study 4) whose claim that no snippet is violation-free no longer holds on SOTorrent22."},{"cited_title":"Stack overflow considered harmful? the impact of copy&paste on android application security,","cited_arxiv_id":null,"evidence_quote":"The 2016 Java crypto-API study whose labeled snippets the authors tracked over seven years, providing the stable comparison case."},{"cited_title":"Stack overflow considered helpful! deep learning security nudges towards stronger cryptography,","cited_arxiv_id":null,"evidence_quote":"The 2018 follow-up Java crypto-API dataset whose labels also remained mostly stable, supporting the niche-topic explanation."},{"cited_title":"Dicos: Discovering insecure code snippets from stack overflow posts by leveraging user discussions,","cited_arxiv_id":null,"evidence_quote":"The DICOS insecure-snippet detection study (case study 2) whose precision and accuracy collapse when re-run on SOTorrent22."},{"cited_title":"Measuring the effects of stack overflow code snippet evolution on open- source software security,","cited_arxiv_id":null,"evidence_quote":"Supplies the NLP classifier used to label post edits as security-relevant for the percentage-of-security-commits (PSC) time series."},{"cited_title":"Snakes in paradise? insecure python-related coding practices in stack overflow,","cited_arxiv_id":null,"evidence_quote":"The Python insecure-practice study (case study 3) whose insecure-answer percentages drop sharply on newer data."},{"cited_title":"(2022) Sotorrent post history ex- tractor","cited_arxiv_id":null,"evidence_quote":"The open-source post-history extractor used to build the SOTorrent22 release once the dataset's authors stopped publishing updates."},{"cited_title":"A study of c/c++ code weaknesses on stack overflow,","cited_arxiv_id":null,"evidence_quote":"The C/C++ weakness study (case study 1) whose CWE-type ranking and revision-improvement conclusions shift on SOTorrent22."}],"review_version":1}