{"id":"86fa46f6-ebcf-4fc9-b7a7-4cd6a807275d","arxiv_id":"2507.17314","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A registered-report protocol proposes the first large-scale map of associations between computational-thinking metrics and code smells in Scratch, but it presents no findings.","lead":"This paper is a registered report: a detailed plan to mine about 2 million public Scratch projects and link nine computational-thinking scores to forty code smells. No empirical results are reported; the text contains research questions, sampling strategy, analysis plans, and validity threats.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"§IV-B sampling premise is unverified: Scratch API offset sampling may not yield uniform coverage of the ≈164M projects, threatening every correlation in the promised map.","rationale":"The reader's weakest assumption coincides with mine: the uniform sampling claim in §IV-B is the single most load-bearing premise. The paper's abstract and Section VII promise a 'first large-scale, fine-grained map' across ~2M projects. If the sample is not representative of the ~164M public projects, every reported correlation, moderation effect, and ML feature ranking is conditional on an unknown selection mechanism. The threat is not merely theoretical: Scratch's public API historically offers listing/search endpoints with limited pagination and no documented guarantee of uniform offsets. The paper cites no documentation or pilot evidence; the Threats to Validity section explicitly asserts, in §V-C, 'we sample ~2M projects uniformly via API' without any calibration. Because a 10-line script can test whether offsets beyond a small bound even succeed, and because a distributional comparison against known creation-time statistics can test representativeness, this concern is both load-bearing and readily settled. I am not urging a verdict change: the paper is a registered report with no empirical results, so UNVERDICTED remains correct until the protocol is either implemented or revised. My concrete test is designed to be run before Stage 1 acceptance. In all other respects the protocol is careful: the power simulation is sound for its target effect, the FDR controls are standard, and the qualitative spot-checks are a good complement. But none of those strengths repair a broken sampling frame.","tokens_in":9313,"tokens_out":3031,"duration_ms":37477,"concrete_test":"Run a pre-registered API probe: issue listing/search requests with offsets 0, 10^3, 10^5, 10^6, 10^7, and 10^8 using the exact endpoint described in §IV-B; record success/error and project IDs. Then compare the created_at distribution of successful offsets against Scratch's cumulative project statistics and against a week-long crawl of the default listing. If any large offset fails, or the created_at distribution deviates significantly from the population curve, uniform random sampling is falsified and the sampling plan must be redesigned (e.g., time-bracketed stratification).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical deliverable is a 'random sample' of ~2M of ~164M public Scratch projects. Section IV-B asserts, without evidence or reference, that 'Random API offsets guarantee an equal inclusion probability.' Public REST APIs commonly cap maximum offset (e.g., 10,000) or apply sort/dedup policies that make high offsets unavailable or non-uniform. If such caps exist, the sample silently collapses to recent or popular projects, and all CT–smell correlations, moderation estimates, and ML results inherit selection bias. The protocol contains no calibration step, and Threats to Validity (§V-C) merely reiterates the assumption. This is load-bearing because external validity of the map depends wholly on representative sampling; a biased sample would make the headline claim uninterpretable. It is also cheap to falsify, so it should be verified prior to Stage-1 acceptance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a Stage-1 registered report for a large-scale repository-mining study. The proposed method is to sample approximately two million public Scratch projects through the Scratch REST API, analyze each project with DrScratch (nine computational-thinking dimensions, four smell indicators) and LitterBox (29 additional smell indicators), and then produce a statistical map of associations between computational-thinking (CT) competencies and code smells. The study is organized around three research questions: an aggregate association between overall CT score and overall smell density (RQ-1), dimension–smell specificity via mutual information and generalized linear models with FDR control and SHAP explanations (RQ-2), and task-context moderation by coding challenge vs. generic projects (RQ-3). The protocol also includes descriptive statistics, exploratory machine learning, a qualitative mini-case study, a power simulation, cross-validation claims, and an open replication package. No empirical results are presented; all claims about the study's contribution are prospective, and the scope note explicitly disclaims causal and longitudinal interpretation.","tokens_in":9423,"tokens_out":7565,"duration_ms":74961,"significance":"If the protocol executes as planned, the study would deliver a genuinely novel empirical artifact: a fine-grained, statistically powered map between nine CT dimensions and an extensive catalogue of design-level smell indicators in block-based novice code, together with effect-size benchmarks and an open pseudonymized dataset. Strengths of the manuscript include the explicit registered-report format with pre-stated hypotheses and a detailed analysis pipeline, a concrete power simulation justifying the target sample size, FDR control and cross-validation plans, the use of two independent linter families with version pinning, and a clear non-causal scope statement. The proposed moderation analysis (RQ-3) is a thoughtful way to separate learner-driven smells from task-induced patterns, and the replication package commitment is appropriate for this venue. However, the central external-validity premise—that random API offsets yield a uniform sample of the ~164M public projects—is asserted without calibration evidence, and the tooling design creates a circularity risk because DrScratch supplies both the CT scores and part of the smell counts.","major_comments":[{"comment":"The load-bearing sampling premise is unverified. The text states that 'Random API offsets guarantee an equal inclusion probability for each of the ≈164 M public projects,' but no evidence, reference, or calibration study is provided. Public search APIs commonly cap the maximum offset, apply implicit sort orders (e.g., by popularity or share date), truncate the index, or deduplicate results; any of these would silently reduce the effective sampling frame to a biased subset (e.g., recent or popular projects) and would propagate selection bias into every RQ-1/RQ-2 correlation and every RQ-3 moderation estimate. Section V-C merely reiterates the assumption ('we sample ~2M projects uniformly via API') rather than testing it. The protocol must add a verification step prior to acceptance: for example, (i) empirically probe the API for offset limits and check whether high offsets return results, (ii) compare the distribution of sampled project IDs and created_at timestamps against known population statistics from scratch.mit.edu/statistics, and (iii) specify a fallback sampling frame or a worst-case coverage bound. Without this, the headline claim of a 'random sample' cannot be accepted.","section":"IV-B, V-C"},{"comment":"Instrument overlap between the CT rubric and the smell detectors threatens RQ-1 and RQ-2 with mechanical correlation. DrScratch computes all nine CT scores and also detects four of the smell indicators (Duplicated Scripts, Dead code, Sprite naming, Backdrop naming), and several CT dimensions are defined over the same block features that LitterBox flags as smells (e.g., the Abstraction score reflects custom-block usage while LitterBox detects 'Unused Custom Block'; duplicated/identical blocks plausibly co-vary with the Abstraction and Data Representation sub-scores). If a CT dimension and a smell count share inputs by construction, a significant association is partially a property of the scoring tools rather than of novice skill. The planned mitigations—tool-version pinning, qualitative spot-checks, multi-tool triangulation—do not address this, because DrScratch is the only source of CT scores and the overlap is within that tool. The manuscript should enumerate for each of the nine CT dimensions which block features feed the score, list the smell–CT pairs that share those features, and pre-register a sensitivity analysis that excludes or adjusts for the mechanically overlapping pairs before interpreting the remaining associations as substantive evidence.","section":"IV-C, V-A, Table II"},{"comment":"The statistical framing of RQ-2 makes its primary hypothesis trivially rejectable. The global null H0: I(d;s)=0 and β_d,s=0 for all 360 dimension–smell pairs will be rejected at N≈2M for effects of negligible practical size; the paper's own power analysis shows >99% power for |ρ|=0.05 at α=0.001, and far smaller per-pair effects will survive FDR correction at this sample size. As written, the test outcome is guaranteed to be 'significant' regardless of the data, which would make the promised fine-grained map uninterpretable as evidence for any specific CT–smell link. The protocol should pre-register a minimal practically relevant effect size or benchmark (e.g., a threshold on |β| or variance explained justified by educational relevance, or mutual information above the 95th percentile of its permutation null), and the deliverable should be framed as an effect-size atlas with confidence intervals rather than a significance list.","section":"II (RQ-2), IV-E, IV-F"}],"minor_comments":[{"comment":"The manuscript refers repeatedly to '40 code smell indicators,' but the specification in Section IV-C and the enumeration in Table II give 4 DrScratch smells plus 29 LitterBox smells, totaling 33. Please correct the count or the catalogue and verify that all downstream mentions (abstract, Section I, Section II, Table V) are consistent.","section":"Abstract, I, II, IV-C, Table II"},{"comment":"The title promises a study of 'Skill Growth,' but the scope note states that the design is cross-sectional with no time-series data and that 'development refers to relative proficiency inferred from completed projects.' The title and introduction should be aligned with the actual design (e.g., 'skill attainment' or 'skill level') to avoid promising longitudinal evidence the study cannot deliver.","section":"Title, II (Scope note)"},{"comment":"The quantitative-qualitative plan is inconsistent: Section IV-E commits to a grounded-theory mini-case study on 30 projects (10 per extreme quantile), while Section V-A and Section VI state that ~385 projects will be qualitatively reviewed (at 95% confidence). Please state the single commitment and reconcile the budgets.","section":"IV-E, V-A, VI"},{"comment":"The RQ-3 interaction coefficient is written as 'γ textCT×textContext' in the text; the stray 'text' substrings should be removed and the coefficient should be defined consistently as γ_CT×Context.","section":"IV-E"},{"comment":"The power analysis reports >99% power for N=1,000,000, but the target sample is ~2M. State the power at the target sample size and clarify how the planned FDR correction across 360 tests was or was not incorporated into the simulation.","section":"IV-F"},{"comment":"For reproducibility of the 'random offsets' design, specify the random-number generator, seed, and an audit trail of the offset sequence; also fix the typographical stray apostrophe in 'remixed’s info' and clarify whether only the latest revision per project is stored.","section":"IV-B, IV-F"},{"comment":"Reference [15] (Lodi and Martini) lacks the venue, volume, and page numbers; please complete the bibliographic entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a Stage-1/registered-report submission, so the decisive question is whether the protocol, if executed, can deliver the promised findings in an interpretable way. I believe the sampling-calibration requirement (major comment 1) and the tool-overlap sensitivity analysis (major comment 2) must be made preconditions for in-principle acceptance; both are verifiable and fixable within the manuscript's scope. I would also ask the editor to hold the authors to the 33-vs-40 smell-count inconsistency and the 'growth' title framing, as these are the kind of specification details a registered report should nail down. I have no concerns about authorship or citation practices from the material I reviewed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a study protocol, not a results paper, so judge it as a plan. The genuinely new thing is the scale and the pairing: nine CT dimensions from DrScratch and 40 smells from DrScratch and LitterBox on roughly two million Scratch projects, with task-context moderation. Prior work either looked at smells in small corpora or used one tool; this would deliver the first fine-grained CT–smell map. That is a legitimate extension of the research program, and the expected payoff is real.\n\nWhat it does well: the RQs are explicit, each with hypotheses and an operational plan. The statistical design is concrete — power simulation at N=10^6, Spearman + robust regression, FDR-controlled tests, cross-validation, bootstrapped CIs. The feasibility section is refreshingly realistic about API rate limits, storage, and timeline. The paper is also honest: the scope note explicitly says these are cross-sectional analyses and no causal claims will be made. That discipline deserves credit.\n\nSoft spots, in order of importance. First, the sampling assumption in §IV-B is load-bearing and unverified. The sentence \"Random API offsets guarantee an equal inclusion probability\" is an assertion, not a demonstrated property. If the Scratch API caps offsets, deduplicates, or sorts results in a way that truncates the search index, the \"random\" sample silently becomes a sample of recent or popular projects. Cheaper to falsify than almost anything else in the protocol — they should test it before Stage-1 acceptance, not just assert it. Second, the shared-measurement issue is underplayed. DrScratch produces both the nine CT scores and four of the smell counts, and both tools parse the same blocks. That is a classic common-method risk; the planned triangulation helps, but the paper should directly discuss how much spurious association this could induce. Third, the novice proxy (Scratch's intended educational positioning, since age data are unavailable) is weak but acknowledged. Fourth, there are no results yet, so the contributions are all anticipated. Minor: the title says \"Affect Skill Growth\" but the design is cross-sectional; the scope note corrects it, but the title overpromises.\n\nNone of these sink the plan. The sampling concern is the one I would want resolved before accepting the registered report, and the shared-method issue deserves a paragraph. For a computing-education reader, this is a solid protocol worth engaging with as a serious referee. My call: accept for peer review, with the sampling calibration and a sharper construct-validity discussion as required revisions.","headline":"A clearly written registered report with a serious statistical plan, but the load-bearing sampling assumption is unverified and the CT/smell measurement overlap needs a sharper defense.","tokens_in":9969,"tokens_out":1833,"would_cite":false,"duration_ms":20235,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper aims to produce the first large-scale, fine-grained map between computational-thinking dimensions and code smells in novice Scratch projects.","keywords":["code smells","computational thinking","Scratch","block-based programming","novice programmers","software maintenance","repository mining","correlation analysis"],"falsifier":"A complete enumeration of public Scratch project IDs, obtained by crawling the site directly or via an independent snapshot, compared against the ID histogram of the API-offset sample: any systematic under- or over-representation of ID ranges would demonstrate non-uniform sampling and refute the external-validity premise. Alternatively, if an independent replication using a full crawl finds no negative Spearman correlation between total CT score and normalized smell density at N = 2 million, the root claim would be refuted.","tokens_in":9102,"feed_emoji":"🧩","tokens_out":6944,"duration_ms":74087,"temperature":0.7,"pith_summary":"This is a registered study protocol, not an empirical report. The authors argue that block-based Scratch repositories make it possible to ask, at scale, whether the design flaws known as code smells track the development of computational thinking. They plan to mine roughly two million randomly sampled public Scratch projects, score each with nine computational-thinking dimensions and forty code-smell indicators from two linters, and test whether higher thinking scores predict lower smell density, which thinking skills best predict which smell families, and whether coding challenges change those relationships. If the plan works, the payoff is the first fine-grained, statistically powered atlas of how early design habits and thinking skills align, useful for curricula and automated feedback tools.","feed_headline":"2M Scratch projects will map code smells to thinking skills","feed_subtitle":"Nine computational-thinking dimensions and 40 design flaws will be correlated in the largest Scratch study yet.","key_machinery":"The machinery is the combination of two automated static-analysis tools run on each .sb3 project: a nine-dimension computational-thinking rubric (abstraction, parallelism, logic, synchronisation, flow control, user interactivity, data representation, math operators, motion operators) and a catalogue of forty code-smell indicators, merged with per-project metadata into rows with normalized smell counts per block. The analysis is then carried by Spearman correlations and robust regression for the aggregate question, permutation mutual information and generalized linear models for dimension-smell specificity, and GLM interaction terms for task-context moderation, with FDR-corrected p-values and 5-fold cross-validation.","core_discovery":"On its own terms, the paper's central claim is that a correlation map between computational thinking and code smells is both attainable and informative: with about two million projects, correlations down to |rho| = 0.05 can be detected with high power, and specific CT dimensions (for example, abstraction, flow control) should show distinctive associations with specific smell families (for example, duplicated scripts, busy waiting). The authors also claim that task context, meaning whether a project is a generic creation or a known coding challenge, may moderate these associations, and that separating context-driven smells from skill-driven smells is necessary before interpreting the map pedagogically. The paper states no results; it establishes the design, hypotheses, and expected evidence for the map.","pith_inferences":["Beyond the paper's stated scope, the same cross-sectional dataset could be sliced by project creation date per author to approximate longitudinal trajectories, testing whether smell density falls as an individual produces more projects.","If the promised negative correlation fails to appear at scale, that would itself be informative: it would suggest CT scores and code-smell detectors measure largely separate constructs, which would complicate but not void the pedagogical value of either.","An editor's reading: the map's usefulness as a feedback signal depends on the sampling assumption, so a cheap validation against an independent snapshot of the Scratch corpus would greatly strengthen the eventual conclusions."],"forward_implications":["Educators would gain an evidence-based ranking of which computational-thinking skills most strongly predict which design flaws, allowing lessons and automated feedback to target the highest-yield pairs.","The study would supply effect-size benchmarks from roughly two million projects, giving later intervention studies a baseline for what counts as a meaningful improvement in design quality.","If moderation holds, instructors could distinguish smells caused by challenge difficulty from smells caused by learner misconceptions, avoiding false conclusions about skill from hard tasks.","The open pseudonymised dataset and reproducible pipeline could seed longitudinal studies of skill growth and serve as a benchmark for smell-detection models."],"supporting_citations":[{"why":"Supplies the LitterBox linter that detects the 29 block-specific code-smell indicators.","marker":"[4]"},{"why":"Supplies the DrScratch tool that computes the nine CT scores and four structural code smells.","marker":"[20]"},{"why":"Establishes that code smells appear even in simple Scratch projects, defining the phenomenon the map targets.","marker":"[8]"},{"why":"Provides the controlled-experiment evidence that smells hamper novice comprehension, motivating the study.","marker":"[9]"},{"why":"Defines Scratch's intended educational positioning and the novice programmer population.","marker":"[5]"},{"why":"Documents how flawed patterns propagate through remixing, making design smells consequential in Scratch.","marker":"[6]"},{"why":"Provides the statistical power analysis justifying the target sample size for detecting small correlations.","marker":"[25]"},{"why":"Supplies the repository-mining study design standard that the method follows.","marker":"[22]"}],"fun_headline_variants":["2M Scratch projects link code smells to thinking skills","Code smells vs. thinking skills: 2M Scratch projects examined","Scratch study: 2M projects reveal smell-thinking correlations","Mapping code smells to thinking skills across 2M Scratch projects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that querying the Scratch API with random offsets gives every public project an equal chance of being included, and it offers no calibration check; if the API search index truncates, sorts, or deduplicates results, the entire sample is biased and all correlations inherit that bias.","fun_headline_variants_meta":{"raw":{"variants":["2M Scratch projects link code smells to thinking skills","Code smells vs. thinking skills: 2M Scratch projects examined","Scratch study: 2M projects reveal smell-thinking correlations","Mapping code smells to thinking skills across 2M Scratch projects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000466,"raw_usage":{"total_tokens":2330,"prompt_tokens":958,"completion_tokens":1372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1301}},"tokens_in":574,"tokens_out":1372,"duration_ms":10480,"temperature":1.0,"reasoning_tokens":1301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:51:03.651108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A complete enumeration of public Scratch project IDs, obtained by crawling the site directly or via an independent snapshot, compared against the ID histogram of the API-offset sample: any systematic under- or over-representation of ID ranges would demonstrate non-uniform sampling and refute the external-validity premise. Alternatively, if an independent replication using a full crawl finds no negative Spearman correlation between total CT score and normalized smell density at N = 2 million, the root claim would be refuted.","supporting_citations":[{"cited_title":"Litterbox: A linter for scratch programs,","cited_arxiv_id":null,"evidence_quote":"Supplies the LitterBox linter that detects the 29 block-specific code-smell indicators."},{"cited_title":"Automaticdetectionofbadprogram- ming habits in scratch: A preliminary study,","cited_arxiv_id":null,"evidence_quote":"Supplies the DrScratch tool that computes the nine CT scores and four structural code smells."},{"cited_title":"Smells in block- based programming languages,","cited_arxiv_id":null,"evidence_quote":"Establishes that code smells appear even in simple Scratch projects, defining the phenomenon the map targets."},{"cited_title":"Do code smells hamper novice programming? a controlled experiment on scratch programs,","cited_arxiv_id":null,"evidence_quote":"Provides the controlled-experiment evidence that smells hamper novice comprehension, motivating the study."},{"cited_title":"Scratch: programming for all,","cited_arxiv_id":null,"evidence_quote":"Defines Scratch's intended educational positioning and the novice programmer population."},{"cited_title":"Remixing as a pathway to computational thinking,","cited_arxiv_id":null,"evidence_quote":"Documents how flawed patterns propagate through remixing, making design smells consequential in Scratch."}],"review_version":1}