pith. machine review for the scientific record. sign in

arxiv: 2604.22120 · v1 · submitted 2026-04-23 · 💻 cs.HC

Recognition: unknown

Same Project, Different Start: How Contribution Events Shape Activity and Retention in Open Source

Mohamed Ouf , Mariam Guizani
Authors on Pith no claims yet

Pith reviewed 2026-05-09 20:16 UTC · model grok-4.3

classification 💻 cs.HC
keywords open sourcecontributor retentioncontribution eventsmentorship programsengagement patternsmatched cohort studynewcomer activityretention factors
0
0 comments X

The pith

Contribution events like mentorship programs increase newcomers' odds of becoming core contributors and extend their retention in open source projects.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates whether organized contribution events produce more lasting contributors than organic entry into open source projects. Using a matched-cohort study of 4,002 contributors across 330 projects, it shows that event-based starters have better chances of reaching core status and remain active for more months. Different entry types lead to distinct patterns of activity in the first 12 weeks, with steady engagement linked to the longest retention but also revealing a drop-off for those who relied on mentorship support. This matters to project maintainers because sustaining contributors is critical, and the early weeks set the trajectory for whether someone stays or leaves.

Core claim

Event contributors have significantly higher odds of becoming core contributors (12.1% vs. 9.6%, p < 0.001, OR = 1.31) and stay significantly longer (median 8.2 vs. 4.8 months). Each entry mechanism is associated with a fundamentally different engagement rhythm: 68.9% of mentorship contributors sustain Steady weekly activity across their first 12 weeks, whereas 61.0% of non-mentorship contributors exhibit Front-Loading and 57.0% of organic contributors exhibit Intermittent engagement. Steady engagement is associated with significantly longer retention regardless of group (median 13 vs. 8 months for Front-Loading), yet mentorship contributors who lose their program scaffolding show shorter

What carries the argument

The matched-cohort comparison of 2,001 event-based contributors with 2,001 organic contributors, together with the identification of three engagement rhythms—Steady, Front-Loading, and Intermittent—based on activity during the initial 12 weeks.

If this is right

  • Event contributors have higher odds of becoming core contributors and longer retention than organic contributors.
  • Entry mechanisms produce different engagement rhythms, with mentorship favoring steady activity.
  • Steady engagement predicts longer retention no matter how the contributor entered the project.
  • Mentorship participants become dependent on program support and retain less after it ends.
  • The first 12 weeks of activity strongly indicate long-term retention outcomes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Open source projects could design contribution events to promote steady engagement patterns to maximize contributor retention.
  • Adding post-program transition support for mentorship participants might counteract the mentor-dependency effect.
  • The influence of entry mechanisms on activity rhythms may extend to other types of collaborative software development communities.
  • Targeting the first 12 weeks with specific onboarding could be an effective strategy for improving overall contributor sustainability.

Load-bearing premise

The matched-cohort design successfully balances all relevant confounders between event-based and organic contributors so that observed differences in retention and engagement can be attributed to the entry mechanism rather than pre-existing differences.

What would settle it

Repeating the matched-cohort analysis on a separate set of open source projects and finding no significant difference in core contributor rates or retention times between event-based and organic groups.

Figures

Figures reproduced from arXiv: 2604.22120 by Mariam Guizani, Mohamed Ouf.

Figure 1
Figure 1. Figure 1: Six-dimensional activity profiles. Each axis shows [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Kaplan-Meier survival curves. (a) Retention: event [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
read the original abstract

Open source projects depend on newcomers who stay, yet most leave after a single contribution. Contribution events such as Google Summer of Code, LFX Mentorship, Hacktoberfest, and 24 Pull Requests attract thousands of newcomers each year, but whether they produce lasting contributors remains unclear. We conduct the first matched-cohort study comparing 2,001 event-based and 2,001 organic contributors across 330 projects. Our results reveal three key findings. First, event contributors have significantly higher odds of becoming core contributors (12.1% vs. 9.6%, p < 0.001, OR = 1.31) and stay significantly longer (median 8.2 vs. 4.8 months). Second, each entry mechanism is associated with a fundamentally different engagement rhythm: 68.9% of mentorship contributors sustain Steady weekly activity across their first 12 weeks, whereas 61.0% of non-mentorship contributors exhibit Front-Loading and 57.0% of organic contributors exhibit Intermittent engagement (p < 0.001). Third, Steady engagement is associated with significantly longer retention regardless of group (median 13 vs. 8 months for Front-Loading), yet mentorship contributors who lose their program scaffolding show shorter retention than self-sustained non-mentorship contributors, revealing a mentor-dependency effect. A newcomer's first 12 weeks are strongly indicative of their long-term trajectory.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper reports a matched-cohort study of 2,001 event-based contributors (from programs such as Google Summer of Code, LFX Mentorship, Hacktoberfest) and 2,001 organic contributors across 330 open-source projects. It claims three main results: (1) event contributors show higher odds of becoming core contributors (12.1% vs. 9.6%, OR = 1.31, p < 0.001) and longer median retention (8.2 vs. 4.8 months); (2) entry mechanisms produce distinct first-12-week engagement rhythms (68.9% Steady for mentorship, 61.0% Front-Loading for non-mentorship, 57.0% Intermittent for organic, p < 0.001); (3) Steady engagement predicts longer retention overall, but mentorship participants exhibit shorter retention once program scaffolding ends, indicating a mentor-dependency effect. The first 12 weeks are presented as strongly predictive of long-term trajectory.

Significance. If the attribution of differences to entry mechanism survives scrutiny for residual selection, the work supplies the first large-scale quantitative evidence on how structured contribution events affect newcomer activity and retention in open source. The concrete statistics (odds ratios, median times, exact percentages) and the identification of engagement-pattern heterogeneity and mentor-dependency provide actionable guidance for program design. The matched-cohort approach is a methodological strength relative to prior smaller or unmatched studies.

major comments (2)
  1. [Methods section on cohort construction and matching] The central attribution of higher core-contributor odds (OR = 1.31) and longer retention (8.2 vs. 4.8 months) to the entry mechanism depends on the matched-cohort design successfully balancing confounders. Matching is described as performed on project, timing, and activity volume, yet no balance tables, propensity-score overlap diagnostics, or sensitivity analyses for unmeasured confounding (e.g., Rosenbaum bounds for baseline motivation or skill) are referenced. Self-selection into events remains a plausible alternative explanation for the observed differences. This issue is load-bearing for the causal interpretation of the three main findings.
  2. [Results section on engagement rhythms] The second and third findings rest on the classification of contributors into Steady, Front-Loading, and Intermittent engagement patterns (with the reported percentages 68.9%, 61.0%, and 57.0%). The manuscript must specify the exact operational criteria, thresholds, or clustering procedure used to assign these labels, together with any validation or sensitivity checks to alternative definitions. Without this detail, the claim that each entry mechanism produces a “fundamentally different engagement rhythm” cannot be fully evaluated.
minor comments (2)
  1. [Abstract] The abstract states that matching was performed but does not enumerate the precise matching variables or mention any robustness checks; adding one sentence on these points would improve transparency.
  2. [Results] A summary table collating the key statistics (core-contributor percentages, OR, median retention times, engagement-pattern percentages, and associated p-values) would aid readers in comparing the three groups.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. We address each major comment point by point below, indicating the revisions we will make to improve transparency and robustness.

read point-by-point responses

  1. Referee: The central attribution of higher core-contributor odds (OR = 1.31) and longer retention (8.2 vs. 4.8 months) to the entry mechanism depends on the matched-cohort design successfully balancing confounders. Matching is described as performed on project, timing, and activity volume, yet no balance tables, propensity-score overlap diagnostics, or sensitivity analyses for unmeasured confounding (e.g., Rosenbaum bounds for baseline motivation or skill) are referenced. Self-selection into events remains a plausible alternative explanation for the observed differences. This issue is load-bearing for the causal interpretation of the three main findings.

    Authors: We agree that the matched-cohort design is central to attributing differences to entry mechanisms and that additional diagnostics are needed to evaluate residual confounding. Our original matching on project, timing, and activity volume was chosen to balance observable factors, but we did not report balance tables, overlap diagnostics, or sensitivity analyses. In the revised manuscript we will add standardized balance tables (pre- and post-matching means and standardized differences), propensity-score overlap plots, and a Rosenbaum bounds sensitivity analysis quantifying the degree of unmeasured confounding (such as baseline motivation or skill) required to overturn the observed odds ratio and retention differences. These additions will directly address the concern about self-selection and strengthen the causal interpretation. revision: yes

  2. Referee: The second and third findings rest on the classification of contributors into Steady, Front-Loading, and Intermittent engagement patterns (with the reported percentages 68.9%, 61.0%, and 57.0%). The manuscript must specify the exact operational criteria, thresholds, or clustering procedure used to assign these labels, together with any validation or sensitivity checks to alternative definitions. Without this detail, the claim that each entry mechanism produces a “fundamentally different engagement rhythm” cannot be fully evaluated.

    Authors: We thank the referee for noting the need for greater detail on the engagement-pattern classification. The patterns were obtained by applying a clustering procedure to the normalized weekly contribution time series over the first 12 weeks. In the revised methods section we will explicitly describe the clustering algorithm, the precise operational criteria and thresholds used to label each pattern, and the results of validation and sensitivity checks (including alternative definitions and time windows). This expanded description will enable readers to fully assess the reported differences in engagement rhythms across entry mechanisms. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical matched-cohort results derive directly from observed data

full rationale

The paper reports a matched-cohort study of 4002 contributors, with results obtained via direct statistical comparisons (odds ratios, p-values, median retention times) on activity logs and engagement patterns. No derivations, equations, fitted predictions, or self-citations appear in the load-bearing steps; the central claims rest on observable differences between event-based and organic groups after matching on project, timing, and volume. This is self-contained empirical analysis with no reduction of outputs to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

This is an observational empirical study with no mathematical derivations. The central claims rest on the validity of cohort matching and the post-hoc classification of activity into three rhythm types.

axioms (2)
  • domain assumption Event-based and organic contributors can be matched on observable project and contribution factors to control for confounding.
    The study design explicitly uses matched cohorts to enable comparison.
  • domain assumption Weekly contribution counts over the first 12 weeks reliably distinguish Steady, Front-Loading, and Intermittent patterns.
    The three engagement types are defined and compared using these counts.

pith-pipeline@v0.9.0 · 5560 in / 1459 out tokens · 60519 ms · 2026-05-09T20:16:21.117177+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages · 1 internal anchor

  1. [1]

    Chue Hong

    Felicity Anderson, Julien Sindt, and Neil P. Chue Hong. 2025. Who Do You Think You Are? Creating RSE Personas from GitHub Interactions. arXiv:cs.SE/2510.05390

    work page arXiv 2025

  2. [2]

    Guilherme Avelino, Eleni Constantinou, Marco Tulio Valente, and Alexander Serebrenik. 2019. On the Abandonment and Survival of Open Source Projects: An Empirical Investigation. InProc. 13th ACM/IEEE Int’l Symp. on Empirical Software Engineering and Measurement (ESEM). 1–12. https://doi.org/10.1109/ESEM.2019. 8870181

    work page doi:10.1109/esem.2019 2019

  3. [3]

    Lingfeng Bao, Xin Xia, David Lo, and Gail C. Murphy. 2021. A Large Scale Study of Long-Time Contributor Prediction for GitHub Projects.IEEE Transactions on Software Engineering47, 6 (2021), 1277–1298. https://doi.org/10.1109/TSE.2019. 2918536

    work page doi:10.1109/tse.2019 2021

  4. [4]

    Ann Barcomb, Klaas-Jan Stol, Dirk Riehle, and Brian Fitzgerald. 2019. Why Do Episodic Volunteers Stay in FLOSS Communities?. InProc. 41st Int’l Conf. on Software Engineering (ICSE). 948–959. https://doi.org/10.1109/ICSE.2019.00100

    work page doi:10.1109/icse.2019.00100 2019

  5. [5]

    Ann Barcomb, Klaas-Jan Stol, Dirk Riehle, and Brian Fitzgerald. 2022. Managing Episodic Volunteers in Free/Libre/Open Source Software Communities.IEEE Transactions on Software Engineering48, 1 (2022), 260–277. https://doi.org/10. 1109/TSE.2020.2982627

    work page arXiv 2022

  6. [6]

    Fabio Calefato, Marco Aurelio Gerosa, Giuseppe Iaffaldano, Filippo Lanubile, and Igor Steinmacher. 2022. Will You Come Back to Contribute? Investigating the Inactivity of OSS Core Developers in GitHub.Empirical Software Engineering27, 3 (2022), 76. https://doi.org/10.1007/s10664-021-10012-6

    work page doi:10.1007/s10664-021-10012-6 2022

  7. [7]

    Jinghui Cheng and Jin L.C. Guo. 2019. Activity-Based Analysis of Open Source Software Contributors: Roles and Dynamics. InProc. 12th Int’l Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, 11–18. https://doi.org/10.1109/CHASE.2019.00011

    work page doi:10.1109/chase.2019.00011 2019

  8. [8]

    Eleni Constantinou and Tom Mens. 2017. An Empirical Comparison of Developer Retention in the RubyGems and npm Software Ecosystems.Innovations in Systems and Software Engineering13, 2–3 (2017), 101–115. https://doi.org/10.1007/s11334- 017-0303-4

    work page doi:10.1007/s11334- 2017

  9. [9]

    DigitalOcean. 2014. Hacktoberfest: A Month-Long Celebration of Open Source. https://hacktoberfest.com. Accessed: 2025-11-07

    2014

  10. [10]

    Fabio Ferreira, Luciana Lourdes Silva, and Marco Tulio Valente. 2020. Turnover in Open-Source Projects: The Case of Core Developers. InProc. XXXIV Brazilian Symposium on Software Engineering (SBES). ACM, 447–456

    2020

  11. [11]

    Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks , booktitle =

    Armstrong Foundjem, Ellis E. Eghan, and Bram Adams. 2021. Onboarding vs. Diversity, Productivity and Quality — Empirical Study of the OpenStack Ecosys- tem. InProc. 43rd IEEE/ACM Int’l Conf. on Software Engineering (ICSE). 1033–1045. https://doi.org/10.1109/ICSE43902.2021.00098

    work page doi:10.1109/icse43902.2021.00098 2021

  12. [12]

    Georgios Gousios, Margaret-Anne Storey, and Alberto Bacchelli. 2016. Work Prac- tices and Challenges in Pull-Based Development: The Contributor’s Perspective. 285–296. https://doi.org/10.1145/2884781.2884826

    work page doi:10.1145/2884781.2884826 2016

  13. [13]

    Mariam Guizani, Thomas Zimmermann, Anita Sarma, and Denae Ford. 2022. Attracting and Retaining OSS Contributors with a Maintainer Dashboard. In Proc. 44th IEEE/ACM Int’l Conf. on Software Engineering: Software Engineering in Society (ICSE-SEIS). 36–40

    2022

  14. [14]

    Adriaan Labuschagne and Reid Holmes. 2015. Do Onboarding Programs Work? A Multi-Level Empirical Analysis of the Effect on Mozilla Contributor Activity. IEEE, 381–384. https://doi.org/10.1109/MSR.2015.49

    work page doi:10.1109/msr.2015.49 2015

  15. [15]

    Bin Lin, Gregorio Robles, and Alexander Serebrenik. 2017. Developer Turnover in Global, Industrial Open Source Projects: Insights from Applying Survival Analysis. InProc. 12th IEEE Int’l Conf. on Global Software Engineering (ICGSE). IEEE, 66–72. https://doi.org/10.1109/ICGSE.2017.17

    work page doi:10.1109/icgse.2017.17 2017

  16. [16]

    Fielding, and James D

    Audris Mockus, Roy T. Fielding, and James D. Herbsleb. 2002. Two Case Studies of Open Source Software Development: Apache and Mozilla.ACM Transactions on Software Engineering and Methodology11, 3 (2002), 309–346. https://doi.org/ 10.1145/567793.567795

    work page doi:10.1145/567793.567795 2002

  17. [17]

    Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017. Curating GitHub for Engineered Software Projects.Empirical Software Engineer- ing22, 6 (2017), 3219–3253. https://doi.org/10.1007/s10664-017-9512-6

    work page doi:10.1007/s10664-017-9512-6 2017

  18. [18]

    Mohamed Ouf, Amr Mohamed, and Mariam Guizani. 2026. Do Good, Stay Longer? Temporal Patterns and Predictors of Newcomer-to-Core Transitions in Conventional OSS and OSS4SG.arXiv preprint arXiv:2601.23142(2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  19. [19]

    Mohamed Ouf, Shayan Noei, Zeph Van Iterson, Mariam Guizani, and Ying Zou

  20. [20]

    Conventional OSS.arXiv preprint arXiv:2601.03430(2026)

    An Empirical Analysis of Community and Coding Patterns in OSS4SG vs. Conventional OSS.arXiv preprint arXiv:2601.03430(2026)

    work page arXiv 2026

  21. [21]

    Herbsleb

    Ei Pa Pa Pe-Than, Alexander Nolte, Anna Filippova, Christian Bird, Steve Scallen, and James D. Herbsleb. 2022. What Happens to All These Hackathon Projects? Identifying Factors to Promote Hackathon Project Continuation.Proceedings of the ACM on Human-Computer Interaction6, CSCW1 (2022), 1–30. https: //doi.org/10.1145/3512904

    work page doi:10.1145/3512904 2022

  22. [22]

    Gustavo Pinto, Igor Steinmacher, and Marco Aurélio Gerosa. 2016. More Common Than You Think: An In-Depth Study of Casual Contributors. InIEEE Int’l Conf. Software Analysis, Evolution, and Reengineering (SANER). IEEE, 112–123

    2016

  23. [23]

    Silva, Igor Scaliante Wiese, Daniel M

    Jefferson O. Silva, Igor Scaliante Wiese, Daniel M. Germán, Christoph Treude, Marco Aurélio Gerosa, and Igor Steinmacher. 2020. Google Summer of Code: Student Motivations and Contributions.Journal of Systems and Software162 (2020), 110487. https://doi.org/10.1016/j.jss.2019.110487

    work page doi:10.1016/j.jss.2019.110487 2020

  24. [24]

    Igor Steinmacher, Ana Paula Chaves, Tayana Conte, and Marco Aurélio Gerosa

  25. [25]

    Preliminary Empirical Identification of Barriers Faced by Newcomers to Open Source Software Projects. InProc. 28th Brazilian Symposium on Software Engineering (SBES). IEEE, 51–60

  26. [26]

    Igor Steinmacher, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles

  27. [27]

    Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects. InProc. 18th ACM Conf. Computer Supported Cooperative Work (CSCW). ACM, 1379–1392

  28. [28]

    Hassan, and Kenichi Matsumoto

    Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2019. The Impact of Automated Parameter Optimization on Defect Prediction Models.IEEE Transactions on Software Engineering45, 7 (2019), 683–

    2019

  29. [29]

    https://doi.org/10.1109/TSE.2018.2794977

    work page doi:10.1109/tse.2018.2794977 2018

  30. [30]

    Wenxin Xiao, Hao He, Weiwei Xu, Yuxia Zhang, and Minghui Zhou. 2023. How Early Participation Determines Long-Term Sustained Activity in GitHub Projects?. InProc. 31st ACM Joint European Software Engineering Conf. and Symposium on the Foundations of Software Engineering (ESEC/FSE). 29–41

    2023

  31. [31]

    Hassan, and Naoyasu Ubayashi

    Kazuhiro Yamashita, Shane McIntosh, Yasutaka Kamei, Ahmed E. Hassan, and Naoyasu Ubayashi. 2015. Revisiting the Applicability of the Pareto Principle to Core Development Teams in Open Source Software Projects. InProc. 14th Int’l Workshop on Principles of Software Evolution (IWPSE). ACM, 46–55

    2015

  32. [32]

    Yang Yue, Yi Wang, and David Redmiles. 2022. Off to a Good Start: Dynamic Contribution Patterns and Technical Success in an OSS Newcomer’s Early Career. 196–200. https://doi.org/10.1109/ICSE-Companion55297.2022.9726925

    work page doi:10.1109/icse-companion55297.2022.9726925 2022

  33. [33]

    Minghui Zhou and Audris Mockus. 2012. What Make Long Term Contributors: Willingness and Opportunity in OSS Community. InProc. 34th Int’l Conf. on Software Engineering (ICSE). IEEE, 518–528

    2012

  34. [34]

    Jiaxin Zhu and Jun Wei. 2019. An Empirical Study of Multiple Names and Email Addresses in OSS. InProc. MSR. 409–420. 6

    2019