REVIEW 2 cited by
Sampling in Software Engineering Research: A Critical Review and Guidelines
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.
Forward citations
Cited by 2 Pith papers
-
A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code
A probabilistic score of code-smell propensity in LLM output is validated, used in a causal analysis, and shown to drop when prompts explicitly discourage known smells.
-
Towards a Science of Causal Interpretability in Deep Learning for Software Engineering
The dissertation presents docode, a causal interpretability method for neural code models, and uses a case study to show that some correlations between code properties and model performance are confounded rather than causal.
Discussion (0). Sign in to comment.