{"id":"b1e7340b-162f-4e3d-b537-ca05bce16b75","arxiv_id":"2502.05054","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using commits, issues, and collaboration networks from ten Ethereum repositories, the study shows planned technical upgrades drive pre-event development activity while market events trigger reactive post-event development.","lead":"This paper studied ten major Ethereum code repositories over a decade and compared developer behavior in the 90 days before and after ten major events such as protocol upgrades, market crashes, and the DAO hack. It found that planned technical changes come with a burst of coding before the event and a slowdown after, while market shocks produce more reactive bursts of work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central technical-vs-market dichotomy is never statistically tested; it is inferred from separate pre/post comparisons and visual inspection, leaving the abstract's headline claim unsupported without an interaction test.","rationale":"The paper provides a substantial dataset, a replication package, and a genuine random-event control, which reduces concern about arbitrary event selection. However, the abstract's headline is a comparative claim: technical events are anticipatory, market events are reactive. That claim cannot be read off the per-pair Wilcoxon tests because those tests only compare pre vs post within each repository-event pair; they do not compare event types. The text in Section VI interprets Fig. 5 and effect sizes, but this is post-hoc narrative, and the differential pattern is not tested. The reader's concern about the 90-day window is real, but I find the missing interaction test more load-bearing because it targets the exact dichotomy in the abstract. A concrete regression-based check can settle this with the existing replication package. If the interaction is not significant, the paper's central claim should be weakened to a descriptive observation. I therefore recommend retaining the conditional verdict (UNCHANGED) rather than accepting as-is.","tokens_in":16750,"tokens_out":10340,"duration_ms":108589,"concrete_test":"Using the replication package, fit a mixed-effects negative binomial regression on daily commit counts for each repository and event, with fixed effects for event type (technical vs market), period (pre vs post 90-day window), their interaction, and random intercepts for repository and event. The central claim requires a significant interaction in the predicted direction (technical: pre>post; market: post>pre). If the interaction is non-significant, the abstract's dichotomy is not statistically supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RQ2 tests only H0: commit distribution in the 90 days before equals the 90 days after, separately for each repository-event pair (Table IV). The abstract's central claim is comparative: technical events show pre-event increases and post-event decreases, while market events are reactive. That requires an event-type × period interaction test, which is never reported. Section VI infers the dichotomy from Fig. 5 and narrative interpretation of effect sizes, but the text itself notes negative r-values are common across event types, and some market events show post-event increases while technical events show decreases. These patterns are confounded with calendar time: market events cluster in 2016–2020, technical events in 2020–2023, so ecosystem growth or maturation could produce the same visual pattern. Without a formal interaction model (e.g., negative binomial mixed model with event type, period, and their interaction) or a simpler event-level comparison of post-pre differences between event types, the headline claim is not statistically established. The 90-day window issue highlighted by the reader compounds this, but even within a fixed window the missing interaction test is the key gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes developer activity in 10 Ethereum repositories (129,884 commits, 40,550 issues) over 2014–2024 to examine how ten major events affect commit activity, issue resolution time, and collaboration networks. The authors report a 90-day impact window, pre/post Wilcoxon tests on commit counts, log-rank survival comparisons for issue resolution, and motif-based network analyses. The headline claim distinguishes technical events, said to prompt pre-event activity and post-event slowdowns, from market events, said to induce reactive development. A replication package is provided.","tokens_in":16967,"tokens_out":5559,"duration_ms":54579,"significance":"If the central claims held, the study would offer a useful longitudinal characterization of blockchain OSS dynamics and a practical template for event-window estimation in such ecosystems. The paper has notable strengths: a large multi-repository dataset, a public replication package, multiple complementary analyses, and a random-event control that supports the specificity of the selected events over arbitrary time points. However, the central technical-vs-market dichotomy is never formally tested, and the 90-day window is derived from the same commit series used for the main tests. These issues currently limit the strength of the conclusions.","major_comments":[{"comment":"The central claim that technical events produce pre-event increases and post-event decreases while market events are reactive is not established by the reported tests. RQ2 tests only the pairwise null that the 90-day pre- and post-event commit distributions are equal for each repository-event pair (Section VI, Table IV); no test compares event types or includes an event-type by period interaction. Figure 5 and the narrative in Section VI are used to infer the dichotomy, but the text itself notes that negative r-values are common across event types and that some market events show post-event increases while technical events show decreases. Because market events cluster in 2016–2020 and technical events in 2020–2023, the visual pattern is also confounded with ecosystem growth and maturation. A formal interaction model (e.g., a negative binomial mixed model with event type, period, and their interaction) or an event-level comparison of post-pre differences between technical and market events should be reported before the abstract's headline claim is made.","section":"Section VI and Abstract"},{"comment":"The 90-day impact window is not independently validated and appears to be derived from the same commit series later used to test event effects. The ACF of commits (Fig. 3) shows a significant positive lag at 1 month and a significant negative lag at 4 months; the text states that normal activity resumes by the fourth month, but a negative lag at month four means commit activity is below its baseline at that lag, not that it has normalized. This contradiction is material because the fixed 90-day pre/post window is applied to all events and metrics in RQ2–RQ4, and the random-event control in Section VI only validates that the selected events differ from random periods, not that the window length is correct. The window should be treated as a sensitivity parameter, with results reported for alternative windows (e.g., 30, 60, 120 days) or estimated separately for each event type or repository.","section":"Section V"},{"comment":"RQ4's network conclusion is based on three events and qualitative inspection of Z-score plots, with no statistical test reported. Section VIII selects the COVID-19 crash, London Hard Fork, and Arrow Glacier Update and asserts from Fig. 9 that Go-ethereum shows significant drops in open triads and spikes in triangles, but no confidence intervals, significance thresholds, or multiple-comparison corrections are provided for the motif Z-scores. With only three events, the claim that technical events enhance core team collaboration cannot be distinguished from repository-specific or calendar-time effects. Please either report a quantitative comparison across more events or rephrase the RQ4 answer as descriptive rather than causal.","section":"Section VIII"},{"comment":"The survival analysis in RQ3 compares cohorts split by issue opening time relative to the event, which does not isolate event effects. Issues opened before an event have longer potential follow-up and are exposed to different repository-management eras than issues opened after; the log-rank test in Table V is therefore sensitive to censoring and global time trends, not only to the event itself. The fact that Go-ethereum shows significant before/after differences for every event in Table V is consistent with a general improvement or backlog-clearing trend over the decade rather than event-specific impacts. A more appropriate design would align issues to event time and compare resolution in symmetric windows, with controls for issue creation rate and repository age, or explicitly model secular trends.","section":"Section VII"}],"minor_comments":[{"comment":"The description of Hardhat (“Contains the specifications for Ethereum 2…”) appears to be a copy-and-paste of the Consensus-Specs row; Hardhat is a development framework, not a specification repository, so this should be corrected.","section":"Table I"},{"comment":"The sentence “we define the post-event window as the third month after an event” is ambiguous; specify whether the post-event window is days 1–90 or days 61–90 after the event.","section":"Section V"},{"comment":"Table IV reports only “x” markers for significant Wilcoxon tests; adding p-values and effect sizes for each repository-event pair, or pointing explicitly to the corresponding table in the replication package, would let readers assess magnitude and direction rather than relying on the narrative.","section":"Table IV"},{"comment":"The caption of Fig. 5 should define how normalized counts are computed and what the circle and cross marks aggregate (e.g., all repositories or per-repository means), because the current figure is difficult to interpret without this information.","section":"Figure 5"},{"comment":"The text says “Using the list of the major events from RQ2,” but RQ2 does not produce an event list; the events are defined in Table II, so please cite Table II here.","section":"Section VII"},{"comment":"The statement that MetaMask is “the largest repository by total activity” is not clearly supported by Table I; Solidity has more commits, and MetaMask has the most issues and comments, so the measure of activity should be defined explicitly.","section":"Section VIII"},{"comment":"The effect-size formula r = Z/√N is reported without stating whether N is the number of paired days or the number of commits; please clarify the definition in the text.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable descriptive longitudinal study, and the replication package is a genuine strength. However, the headline technical-versus-market dichotomy overreaches the current analysis: the key interaction is missing, the 90-day window is circularly derived from the same commit data, and the network and survival results are confounded with time trends. These issues are addressable with additional analysis rather than being unfixable, so I recommend major revision rather than rejection. If the authors add the interaction test, sensitivity analyses for the window, and a more defensible survival or network design, the paper could make a solid contribution to the mining-software-repositories literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a useful descriptive longitudinal study of Ethereum developer activity around major events, and the replication package is a real asset. What's genuinely new is the combination of commit, survival, and network-motif analyses across ten repositories and ten events; no prior work in the cited literature does that. The random-event control (100 random events, zero significant results after correction) is a legitimate check that the selected events are not just noise. The survival analysis produces plausible, repository-specific patterns, and the threats-to-validity section is honest. The soft spot is load-bearing: the abstract's central claim that technical events cause pre-event surges and post-event dips, while market events cause reactive bursts, is never statistically tested. RQ2 compares commit distributions in the 90 days before vs. after each event, separately for each repository-event pair. That does not test whether the pre-post difference varies by event type. The stress-test note is right: an interaction model (event type x period) or at least a comparison of post-pre differences across event types is missing. The issue is compounded by confounding with calendar time: market events cluster in 2016-2020, technical events in 2020-2023, so ecosystem maturation alone could produce the same visual pattern. The 90-day window is also fitted from the same commit series used in the main tests. The ACF shows a significant negative lag at four months, which actually contradicts the claim that activity normalizes by month four. That is a circularity concern, and a sensitivity analysis over window length would help. RQ4 is more exploratory: only three events, with conclusions drawn from visual inspection of motif Z-scores, so the claim that technical events enhance core team collaboration is suggestive at best. Overall, this is a solid empirical study that overreaches in its headline. The data and descriptive findings deserve to see the light, but the comparative claim needs a proper interaction test and a treatment of the temporal confound. I would send it to peer review with major revision expectations, not desk reject it. The paper is for empirical software engineering and blockchain ecosystem researchers, and the replication package makes it worth engaging with seriously.","headline":"A competent descriptive mining study with a solid data package, but the headline technical-vs-market dichotomy is asserted rather than tested.","tokens_in":622,"tokens_out":1039,"would_cite":false,"duration_ms":26143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ten years of Ethereum data show technical events slow commits, market events speed them.","keywords":["Ethereum","open source software","developer activity","event impact","commit analysis","survival analysis","collaboration networks","longitudinal study"],"falsifier":"Recompute the commit and issue analyses with event-specific windows selected by change-point detection or by the data itself; if the pre-event build-up and post-event dip for technical events disappears or reverses under variable windows, the central claim fails. Alternatively, a larger random-event control that produced as many significant results as the real event set would falsify the event-driven interpretation.","tokens_in":16572,"feed_emoji":"📉","tokens_out":4487,"duration_ms":43636,"temperature":0.7,"pith_summary":"This paper tries to establish that major events in Ethereum's history change how its developer community works, and that the direction of change depends on event type. Over ten years and ten repositories, technical upgrades like The Merge draw extra commits before the event and fewer afterward, while market shocks like the COVID-19 crash trigger reactive commit bursts. The authors argue this distinction matters because it reveals whether open-source development is planned or reactive, and because it gives maintainers a concrete 90-day window for expecting disruption. A sympathetic reader would take this as evidence that blockchain open-source communities respond to external shocks in structured, measurable ways.","feed_headline":"Technical upgrades slow Ethereum commits; market shocks speed them","feed_subtitle":"Ten-year study of 10 repositories finds event effects last 90 days and differ by event type.","key_machinery":"The central object is the 90-day pre/post event window, derived from autocorrelation analysis of monthly commit counts and then applied uniformly to all events and metrics. It is paired with a repository-event analysis in which each of ten repositories is treated as an independent experiment group; Wilcoxon signed-rank tests with Benjamini-Hochberg false-discovery correction compare commit distributions before and after each event. For issues, Kaplan-Meier survival curves and log-rank tests compare resolution times; for collaboration, weighted contributor networks are built from co-commenting and examined through motif Z-scores against a configuration-model null. The 90-day window is what carries the argument: it defines the \"before\" and \"after\" for every test.","core_discovery":"The central claim is that technical events prompt increased activity before the event, followed by reduced commit rates afterwards, whereas market events lead to more reactive development. The authors find this pattern by comparing 90-day windows before and after ten major events across ten repositories, with statistical tests showing significant commit-activity changes for events such as the Frontier Release, the DAO hack, and the Beacon Chain Launch. They also report that event effects on developer collaboration last about 90 days, that issue resolution times shift significantly around most events, and that technical events tighten collaboration among core teams while the 2020 market crash had the smallest structural impact. If true, the paper establishes a temporal and typological account of how external events perturb developer activity in a cryptocurrency ecosystem.","pith_inferences":["The fixed 90-day window may conceal event-specific horizons; an event-wise change-point or multi-horizon analysis could reveal that market shocks act faster and technical upgrades linger longer.","The pre-event commit build-up pattern could be used as a low-cost leading indicator: a surge in core-repository commits before an announced protocol change may predict where compatibility work will land.","The motif analysis suggests a testable extension: if technical events reliably increase closed triads in core repositories, triangle density could be monitored as an early-warning signal for coordination load.","Because the data cover one ecosystem, the planned-versus-reactive distinction is best treated as a candidate hypothesis for other token-bearing open-source projects, not a settled law."],"forward_implications":["Maintainers should expect roughly three months of altered activity after any major event, with normal patterns returning by the fourth month.","For planned technical upgrades, preparation work should be scheduled before the event, since commit rates tend to drop afterward.","For market shocks, teams should anticipate reactive commit bursts and allocate review and merge capacity accordingly.","Core infrastructure repositories such as Go-Ethereum resolve issues faster and show stronger collaboration tightening during technical events, so they may serve as early indicators of ecosystem stress.","Event effects are not random: 100 random events produced no significant commit changes, so the observed patterns are tied to the selected major events."],"supporting_citations":[{"why":"Supplies the Shapiro-Wilk normality test used to justify the non-parametric Wilcoxon signed-rank tests for commit comparisons.","marker":"[38]"},{"why":"Provides the Benjamini-Hochberg procedure used to control the false discovery rate across multiple repository-event tests.","marker":"[39]"},{"why":"Supplies the configuration-model null and Z-score approach used to assess motif significance in collaboration networks.","marker":"[44]"},{"why":"Defines the issue resolution time metric that the survival analysis in RQ3 is built on.","marker":"[41]"},{"why":"Links issue resolution time to team responsiveness and operational challenge, motivating RQ3's interpretation.","marker":"[17]"},{"why":"Provides the prior event-impact analysis of unusual GitHub activity that this study adapts to blockchain events.","marker":"[35]"},{"why":"Gives a prior event study of Ethereum's proof-of-stake transition, which the paper extends to a broader event set and multi-metric analysis.","marker":"[37]"}],"fun_headline_variants":["Technical upgrades slow Ethereum commits, market shocks speed them","Ethereum devs: event type drives commit rates for 90 days","10-year Ethereum study: tech events reduce commits, market events increase","How Ethereum devs adapt: upgrades vs. market crashes","Ethereum events: technical vs. market impacts on contributor activity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single fixed 90-day window captures the full effect of every major event; if effects actually last longer or shorter for some events or repositories, the before/after comparisons built on that window could misstate the impact.","fun_headline_variants_meta":{"raw":{"variants":["Technical upgrades slow Ethereum commits, market shocks speed them","Ethereum devs: event type drives commit rates for 90 days","10-year Ethereum study: tech events reduce commits, market events increase","How Ethereum devs adapt: upgrades vs. market crashes","Ethereum events: technical vs. market impacts on contributor activity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1779,"prompt_tokens":812,"completion_tokens":967,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":879}},"tokens_in":428,"tokens_out":967,"duration_ms":8717,"temperature":1.0,"reasoning_tokens":879,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T20:24:38.642974+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the commit and issue analyses with event-specific windows selected by change-point detection or by the data itself; if the pre-event build-up and post-event dip for technical events disappears or reverses under variable windows, the central claim fails. Alternatively, a larger random-event control that produced as many significant results as the real event set would falsify the event-driven interpretation.","supporting_citations":[{"cited_title":"An analysis of variance test for normality (complete samples),","cited_arxiv_id":null,"evidence_quote":"Supplies the Shapiro-Wilk normality test used to justify the non-parametric Wilcoxon signed-rank tests for commit comparisons."},{"cited_title":"Quick and easy im- plementation of the benjamini-hochberg procedure for controlling the false positive rate in multiple comparisons,","cited_arxiv_id":null,"evidence_quote":"Provides the Benjamini-Hochberg procedure used to control the false discovery rate across multiple repository-event tests."},{"cited_title":"Early-warning signals of topological collapse in interbank networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the configuration-model null and Z-score approach used to assess motif significance in collaboration networks."},{"cited_title":"Are bullies more productive? empirical study of affective- ness vs. issue fixing time,","cited_arxiv_id":null,"evidence_quote":"Defines the issue resolution time metric that the survival analysis in RQ3 is built on."},{"cited_title":"An empirical study of speed and communi- cation in globally distributed software development,","cited_arxiv_id":null,"evidence_quote":"Links issue resolution time to team responsiveness and operational challenge, motivating RQ3's interpretation."},{"cited_title":"Unusual events in github reposi- tories,","cited_arxiv_id":null,"evidence_quote":"Provides the prior event-impact analysis of unusual GitHub activity that this study adapts to blockchain events."},{"cited_title":"An event study of the ethereum transition to proof-of-stake,","cited_arxiv_id":null,"evidence_quote":"Gives a prior event study of Ethereum's proof-of-stake transition, which the paper extends to a broader event set and multi-metric analysis."}],"review_version":1}