Recognition: unknown
Good Question! The Effect of Positive Feedback on Contributions to Online Public Goods
Pith reviewed 2026-05-10 14:59 UTC · model grok-4.3
The pith
Randomly assigning an anonymous upvote to new Stack Overflow questions increases the author's later contributions to the site.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
In a randomized experiment on Stack Overflow with over 22,000 new questions, receiving one anonymous upvote raised the probability that the author would post another question by 6.3 percent and answer someone else's question by 12.9 percent within four weeks. A second upvote added no further effect. The increase in answering was larger and remained detectable at twelve weeks. Visibility gains from the upvote played little role in the rise in new questions but largely accounted for the rise in answering, because higher-ranked questions attracted more answers and that experience appeared to move the original poster toward wider participation.
What carries the argument
Randomized assignment of an anonymous upvote to isolate the effect of perceived positive feedback from changes in question visibility or other platform actions.
If this is right
- A single anonymous upvote raises both asking and answering rates, but the answering effect is larger and lasts longer.
- Algorithmic visibility explains most of the gain in answering but almost none of the gain in asking new questions.
- Receiving answers on one's own question, triggered by the initial visibility boost, shifts users toward answering others' questions.
- Additional upvotes beyond the first produce no measurable extra contribution.
Where Pith is reading between the lines
- Platforms could test low-cost positive signals to slow declines in volunteer activity across other public-good sites.
- The pattern suggests a feedback loop in which an early answer creates habit or belonging that sustains wider contributions.
- The same design could be applied to measure whether different feedback types, such as comments or badges, produce comparable shifts.
- Effects may vary with a user's prior activity level or the technical difficulty of the original question.
Load-bearing premise
Recipients interpret the anonymous upvote as a genuine positive signal about their own question rather than as random or automatic platform noise.
What would settle it
A follow-up trial in which the same upvote is assigned but recipients receive no notification or are told the vote is automatic would show no rise in subsequent contributions.
Figures
read the original abstract
Online platforms where volunteers answer each other's questions are important sources of knowledge, yet participation is declining. We ran a pre-registered experiment on Stack Overflow, one of the largest Q&A communities for software development (N = 22,856), randomly assigning newly posted questions to receive an anonymous upvote. Within four weeks, treated users were 6.3% more likely to ask another question and 12.9% more likely to answer someone else's question. A second upvote produced no additional effect. The effect on answering was larger, more persistent, and still significant at twelve weeks. Next, we examine how much of these effects are due to algorithmic amplification, since upvotes also raise a question's rank and visibility. Algorithmic amplification is not important for the effect on asking additional questions, but it matters a lot for the effect on answering other questions. The increase in visibility increases the probability that another user provides an answer, and that experience appears to shift the poster toward broader community participation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a pre-registered field experiment on Stack Overflow (N=22,856) that randomly assigns an anonymous upvote to newly posted questions. It claims that within four weeks, treated users are 6.3% more likely to ask another question and 12.9% more likely to answer someone else's question, with the answering effect larger, more persistent, and significant at twelve weeks. A second upvote adds no effect. The paper further decomposes these into positive-feedback versus algorithmic-amplification channels, concluding that amplification is unimportant for additional questions but drives much of the effect on answering other questions via increased visibility and subsequent experience.
Significance. If the results hold, the work supplies clean causal evidence on how minimal positive signals and visibility changes affect repeated contributions to online public goods, with direct implications for platform design. Notable strengths include the large sample, pre-registration, random assignment of a single treatment, measurement of both short- and twelve-week outcomes, and an explicit mechanism test separating feedback from amplification.
major comments (2)
- [mechanism analysis / abstract] The interpretation that the observed increases in future contributions are produced by 'positive feedback' (abstract and mechanism analysis) rests on the assumption that recipients perceive the experimenter-assigned anonymous upvote as genuine community approval rather than platform noise. Randomization identifies the effect of receiving the upvote but supplies no direct evidence (surveys, click data, or behavioral proxies) that users notice or correctly attribute the upvote. This perception step is load-bearing for both headline effects and for the claim that algorithmic amplification (rather than feedback per se) accounts for the answering result.
- [mechanism analysis] The decomposition of the answering effect into amplification versus feedback channels (abstract) compares outcomes across conditions but does not report robustness checks for differential attrition, compliance with the upvote delivery, or platform-side responses that could confound the visibility channel. Without these, the quantitative attribution of 'how much' is due to amplification remains difficult to evaluate.
minor comments (2)
- [abstract / methods] The abstract and main text should report exact sample sizes after any exclusions, attrition rates between four and twelve weeks, and compliance rates for upvote delivery.
- [mechanism analysis] Clarify the precise operationalization of 'algorithmic amplification' (e.g., changes in question rank, views, or answer probability) and how it is isolated from other platform responses.
Simulated Author's Rebuttal
We thank the referee for their constructive and detailed comments. We respond to each major comment below and indicate the revisions we will make.
read point-by-point responses
-
Referee: [mechanism analysis / abstract] The interpretation that the observed increases in future contributions are produced by 'positive feedback' (abstract and mechanism analysis) rests on the assumption that recipients perceive the experimenter-assigned anonymous upvote as genuine community approval rather than platform noise. Randomization identifies the effect of receiving the upvote but supplies no direct evidence (surveys, click data, or behavioral proxies) that users notice or correctly attribute the upvote. This perception step is load-bearing for both headline effects and for the claim that algorithmic amplification (rather than feedback per se) accounts for the answering result.
Authors: We agree that the positive-feedback interpretation relies on users perceiving the upvote as a community signal rather than noise. The experiment delivers the upvote through Stack Overflow's standard interface, so the question score increases visibly to the poster in the same manner as any other upvote. While we lack direct perceptual data such as surveys or click logs, this delivery method aligns with normal platform feedback. In revision we will (a) rephrase the abstract and mechanism section to state the assumption explicitly and (b) add a limitations paragraph discussing the absence of perceptual measures and the plausibility of the assumption given platform norms. revision: partial
-
Referee: [mechanism analysis] The decomposition of the answering effect into amplification versus feedback channels (abstract) compares outcomes across conditions but does not report robustness checks for differential attrition, compliance with the upvote delivery, or platform-side responses that could confound the visibility channel. Without these, the quantitative attribution of 'how much' is due to amplification remains difficult to evaluate.
Authors: We will add the requested robustness checks. The revised mechanism section will include: (1) tests for differential attrition via balance checks on response rates and observables across arms; (2) documentation of treatment compliance, which was implemented directly by the research team; and (3) auxiliary analyses of platform metrics (e.g., question views) to assess possible confounds in the visibility channel. These additions will support the quantitative claims about the amplification component. revision: yes
- Direct evidence on user perception of the anonymous upvote (surveys, click data, or behavioral proxies), which cannot be collected retroactively because the pre-registered experiment has concluded.
Circularity Check
No circularity in randomized experimental design
full rationale
The paper reports results from a pre-registered randomized controlled trial on Stack Overflow (N=22,856), where newly posted questions are randomly assigned to receive an anonymous upvote, with direct measurement of subsequent user contributions (asking and answering questions) over four and twelve weeks. No equations, fitted models, or derivations are used to produce the headline effects; the 6.3% and 12.9% increases are identified via randomization and outcome tracking. There are no self-definitional steps, fitted inputs called predictions, or load-bearing self-citations that reduce claims to inputs by construction. The analysis of algorithmic amplification versus feedback is also based on the same experimental variation and direct comparisons, remaining self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Random assignment of the upvote isolates the causal effect of receiving positive feedback
- domain assumption No significant spillover or general-equilibrium effects from treating a subset of questions
Reference graph
Works this paper leans on
-
[1]
From asking to answer- ing: Getting more involved on stack overflow.arXiv preprint arXiv:2010.04025,
Timur Bachschi, Anik´ o Hann´ ak, Florian Lemmerich, and Johannes Wachs. From asking to answer- ing: Getting more involved on stack overflow.arXiv preprint arXiv:2010.04025,
-
[2]
Socializing vol- unteers in an online community: A field experiment
Rosta Farzan, Robert Kraut, Laura Dabbish, Tom Postmes, and Hilda De Keyzer. Socializing vol- unteers in an online community: A field experiment. InProceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, pages 325–334,
2012
-
[3]
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
26 Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,
work page internal anchor Pith review arXiv
-
[4]
Follow the (slash) dot: Effects of feedback on new members in an online community
Cliff Lampe and Erik Johnston. Follow the (slash) dot: Effects of feedback on new members in an online community. InProceedings of the 2005 International ACM SIGGROUP Conference on Supporting Group Work, pages 11–20,
2005
-
[5]
Strate- gic behavior and AI training data.arXiv preprint arXiv:2404.18445,
Christian Peukert, Florian Abeillon, J´ er´ emie Haese, Franziska Kaiser, and Alexander Staub. Strate- gic behavior and AI training data.arXiv preprint arXiv:2404.18445,
-
[6]
Data leverage: A framework for empowering the public in its relationship with technology companies
Nicholas Vincent, Hanlin Li, Nicole Tilly, Stevie Chancellor, and Brent Hecht. Data leverage: A framework for empowering the public in its relationship with technology companies. InProceed- ings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 215–227,
2021
-
[7]
Organizing without formal organization: Group identification, goal setting, and social modeling in directing online production
Haiyi Zhu, Robert Kraut, and Aniket Kittur. Organizing without formal organization: Group identification, goal setting, and social modeling in directing online production. InProceedings of the ACM 2013 Conference on Computer Supported Cooperative Work, pages 935–946,
2013
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.