{"id":"8f03953d-2501-47cf-a3e9-ededaf8e2c5a","arxiv_id":"2608.04352","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across Stack Exchange, CS educators' posts are mostly technical and dominated by programming topics, while non-IT topics such as mathematics grew steadily between 2013 and 2018.","lead":"This paper applies topic modeling to millions of posts by users of the CS Educators Stack Exchange site and finds their discussions mix technical programming topics with non-technical subjects such as mathematics and humanities. It provides a descriptive map of what CS educators talk about across the network and how those interests shifted from 2008 to 2024.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 64.88/35.12 IT/Non-IT split rests on an unvalidated, ambiguously defined corpus filter: accepted-answer selection plus an unreconciled count mismatch (748,084 vs 784,048), so the headline percentages may not represent all CS-educator discussions.","rationale":"The reader's verdict is CONDITIONAL, and I find no reason to move away from it. The central claim—that CS educators' Stack Exchange discussions are majority IT but include a substantial non-IT component—is plausible and supported by the paper's descriptive analysis, but its quantitative form (64.88/35.12) depends on a corpus construction that is ambiguous and unvalidated. Section 3.1 does not clearly specify whether accepted answers are filtered by authorship, and the reported dataset size conflicts with Table 1. Because the filtering excludes all unaccepted answers, which constitute a substantial share of posts on Stack Exchange sites, the topic distribution over the remaining posts may not equal the distribution over all CS-educator discussions. No sensitivity analysis or comparison with the unfiltered corpus is provided. This is a correctness risk rather than a fundamental flaw: the finding could survive a robustness check, but until one is performed, the quantitative headline should be treated as conditional. The reader's weakest_assumption aligns with this concern, so my assessment agrees. The concrete test above would resolve it by rerunning the topic model on the unfiltered corpus and comparing category percentages.","tokens_in":9848,"tokens_out":9158,"duration_ms":88660,"concrete_test":"Reconstruct the corpus from the May 2024 Stack Exchange data dump exactly as described: start with all posts whose OwnerUserId maps (via Users.xml) to an AccountId present on the CS Educators site; then apply the two readings of Section 3.1—(i) keep all questions plus accepted answers regardless of answer authorship, and (ii) keep all questions plus accepted answers authored by CS Educators. Re-run the same preprocessing and MALLET LDA with K=55 on each, plus a third corpus containing all posts by these users (no accepted-answer filter). Compare the resulting IT/Non-IT category percentages using the paper's manual topic-to-category mapping. If the 64.88/35.12 split changes by more than 5 percentage points under the unfiltered or alternative-filtered corpus, the headline finding is an artifact of the filter.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is in Section 3.1, where the corpus is defined as posts authored by CS Educators users but then restricted to 'questions and their accepted answers.' The text is ambiguous about whether accepted answers must also be authored by CS educators, and it is contradicted by Table 1: the text reports 748,084 entries (397,061 questions + 351,023 accepted answers), while Table 1 lists 784,048 posts used for topic modeling. If the dataset includes only accepted answers, then each included answer is tied to a question that has an accepted answer; yet 397,061 questions exceed 351,023 accepted answers, so either 46,038 questions lack accepted answers (and were nevertheless kept) or the selection rule is misstated. More importantly, unaccepted answers—potentially a large fraction of CS educators' posts, since the network-wide answer/question ratio is ~1.48—are excluded with no evidence that the topic distribution over the remaining posts matches the full corpus. The reported IT (64.88%) and Non-IT (35.12%) percentages are computed on this filtered subset; if accepted answers are systematically more technical (e.g., because well-defined programming questions are more likely to receive accepted answers), the headline split would overstate IT dominance. The paper's own threats-to-validity section does not discuss this potential selection bias. Without a comparison against the unfiltered corpus, the central quantitative claim is not robust.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes posts by users of the CS Educators Stack Exchange across 169 English-language Stack Exchange sites, using LDA topic modeling (K=55) on a corpus described inconsistently as 748,084 or 784,048 posts, manually labeling topics and grouping them into IT and Non-IT categories. For RQ1, the paper reports that 64.88% of posts are IT-related and 35.12% are Non-IT, with programming and software development dominant in IT and mathematics dominant in Non-IT. For RQ2, it reports absolute and relative impact trends, including a claim of steady Non-IT growth from 2013 to 2018. The paper positions this as the first cross-network analysis of CS educators' discussions on Stack Exchange.","tokens_in":10141,"tokens_out":5260,"duration_ms":49114,"significance":"If the empirical findings withstand scrutiny, the study provides useful descriptive evidence about the breadth of CS educators' participation across Stack Exchange, going beyond prior single-site studies. The cross-network user tracing via AccountId, the large dataset, and the coherence-based selection of the LDA topic count K are notable strengths. However, the central percentages and trend claims are currently undermined by unresolved corpus count inconsistencies, an unvalidated accepted-answer filter, absent inter-rater reliability, and flaws in the relative-impact definition. These issues are fixable, but they are load-bearing for the paper's main claims, so the manuscript requires major revision.","major_comments":[{"comment":"The corpus size and selection rule are inconsistent. The text says the final dataset comprised 748,084 entries (397,061 questions + 351,023 accepted answers), while Table 1 lists 784,048 posts used for topic modeling; since every percentage in RQ1 is computed on this corpus, the discrepancy must be resolved. Moreover, if only questions with accepted answers are retained, the number of questions should equal the number of accepted answers; the 46,038 difference between 397,061 and 351,023 suggests either questions without accepted answers were kept, accepted answers include answers by non-CS-educators, or the selection rule is misstated. Please clarify the exact filtering and report a single corpus count.","section":"Section 3.1, Table 1"},{"comment":"The decision to analyze only accepted answers (and possibly only questions with accepted answers) is not validated. Excluding unaccepted answers removes a large fraction of CS educators' posts, and if accepted answers are more likely to be attached to well-specified technical questions, the reported 64.88% IT share would be biased upward. The paper should compare topic distributions on the full corpus of CS-educator posts, or at least on questions plus all answers, and report whether the IT/Non-IT split is stable under this filter.","section":"Section 3.1, RQ1"},{"comment":"The manual labeling and hierarchical grouping drive all quantitative claims in RQ1 and RQ2, but no inter-rater reliability is reported. The text says 'two raters' and 'over 20 Zoom-based iterations,' yet no agreement metric (e.g., Cohen's kappa) or confusion analysis is given. Please report quantified agreement for topic labels and category assignments, or justify why label noise cannot materially affect the 64.88/35.12 split and the trend results.","section":"Section 3.2, manual labeling"},{"comment":"Equations (4) and (5) are internally inconsistent. Equation (4) defines P(m) as the total number of posts in month m that contain topic x_n; with that definition the ratio is identically 1. The indicator in Equation (4) refers to 'topic x_n is present' whereas Equation (2) defines dominance, so the two metrics are not aligned. Equation (5) sums relative impacts over topics, which may exceed 1 if a post can contain multiple topics. Please correct the definitions and clarify whether relative impact is a proportion of posts whose dominant topic is in the category. In addition, the claim of 'steady growth from 2013 to 2018' in Section 5.1.1 is stated without trend fitting or confidence intervals, so the reader cannot assess whether the growth is meaningful.","section":"Section 5, Equations (4) and (5)"}],"minor_comments":[{"comment":"There is a typo in 'CS Eucators'; it should be 'CS Educators'.","section":"Section 2"},{"comment":"The subcategory percentage for Programming and Software Development is given as 23.00% in Section 4.1 and 23.04% in the Summary of RQ1; also, the listed subcategory percentages for IT and Non-IT do not sum to exactly 100%. Please reconcile the numbers.","section":"Section 4.1 and Summary of RQ1"},{"comment":"References [33] and [34] are the same paper and should be deduplicated.","section":"References"},{"comment":"The figure 79,854,463 refers to the full Stack Exchange corpus, not the analyzed subset; please state this explicitly to avoid confusion with the 748,084/784,048 entries actually used.","section":"Abstract and Introduction"},{"comment":"The figures for absolute and relative impact are not included in the provided manuscript; when finalized, ensure the axes are labeled, the legend is legible, and the lines are clearly identified as raw counts or smoothed trend lines.","section":"Figures 5 and 6"}],"recommendation":"major_revision","confidential_remarks":"This is a descriptive empirical study with a substantial data-collection effort. The central idea is sound, and the main quantitative claims are likely defensible once the corpus definition is corrected and validated. The key blockers are the unresolved count mismatch between 748,084 and 784,048, the unvalidated accepted-answer filter, the missing inter-rater reliability, and the flawed relative-impact definition. These are all fixable within the manuscript's scope, so I recommend major revision rather than rejection. The duplicate reference and numeric inconsistencies suggest the final version needs careful proofreading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as a first-cut descriptive map: someone ran LDA over posts authored by CS Educators users across all English Stack Exchange sites, hand-labeled 55 topics, and grouped them into IT (64.88%) and Non-IT (35.12%). The empirical scope is genuinely new — no prior work tracks this population network-wide — and the hybrid API/data-dump approach to resolve cross-site account IDs is a sensible fix for a real data problem. The method is off-the-shelf LDA, but that's fine for a descriptive study; coherence-based K selection and card-sorting labeling are standard.\n\nThe soft spots are real, and they hit the headline numbers. The corpus definition is sloppy. The text says the final dataset was 748,084 entries (397,061 questions + 351,023 accepted answers), but Table 1 lists 784,048 posts used for topic modeling. Those don't reconcile. The selection rule is also ambiguous: 'selecting only those with accepted answers' could mean only questions that have accepted answers, but then 397k questions vs 351k accepted answers is impossible; or it could mean all questions plus accepted answers, which isn't a coherent filter. More importantly, there is no check against the unfiltered corpus. Accepted answers are a specific subset — they're answers the asker chose. If accepted answers skew toward well-specified technical questions (which is plausible), the IT/Non-IT split is biased. The threats-to-validity section never mentions this. Manual labeling also lacks inter-rater reliability; two raters are mentioned, but no kappa. No code or data are provided. The time-trend claims (Non-IT growth 2013–2018) are visual inspections, not statistical tests.\n\nThe central claim — that CS educators talk mostly about programming, with math as the top non-IT topic — is probably correct in direction. But the paper as written doesn't establish it robustly. All of these problems are fixable: reconcile the counts, clarify the filter, run a sensitivity comparison against the full post set, add inter-rater agreement, release code/data.\n\nI'd send this to a serious referee. It's a plausible new empirical result and the community would benefit from a solid version. But the revision bar should be high, and the dataset questions need to be answered before publication.\n\nRecommendation: engage with it, but require the corrections.","headline":"Plausible first map of CS-educator talk across Stack Exchange, but the corpus definition is sloppy enough that the headline split isn't trustworthy yet.","tokens_in":10646,"tokens_out":4081,"would_cite":false,"duration_ms":36327,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 748,084 posts by CS educators on Stack Exchange, 55 discussion topics emerge, with IT at 64.88% and non-IT at 35.12%, and non-technical topics growing steadily from 2013 to 2018.","keywords":["Stack Exchange","CS educators","topic modeling","Latent Dirichlet Allocation","computer science education","Q&A communities","discussion trends"],"falsifier":"Take a random sample of posts excluded by the accepted-answer filter, apply the same preprocessing and LDA model, and compare the resulting category shares against the reported 64.88% IT / 35.12% Non-IT split; a material shift would show the filter distorted the claimed distribution.","tokens_in":9657,"feed_emoji":"💬","tokens_out":6398,"duration_ms":57224,"temperature":0.7,"pith_summary":"This paper sets out to show what computer science educators talk about when they use the Stack Exchange network, not just on the dedicated CS Educators site but across all 169 English-language sites. By running topic modeling over 748,084 questions and accepted answers authored by CS Educators users, it claims to have mapped a hierarchy of 55 discussion topics. The headline finding is a split: 64.88% of discussion activity falls under IT, led by programming and software development, while 35.12% covers mathematics, humanities, lifestyle, physics, and other non-technical areas. Because the non-IT share grew steadily between 2013 and 2018, the paper argues that CS educators' interests are broadening beyond technical topics, which matters for anyone designing support, training, or tools for this community.","feed_headline":"One in three CS educator posts falls outside IT","feed_subtitle":"A topic-model analysis of 748,084 posts finds math, humanities, and lifestyle talk gaining ground among CS educators.","key_machinery":"The engine of the analysis is Latent Dirichlet Allocation (LDA), a probabilistic topic model run through the MALLET library. The authors preprocess all posts by stripping code and HTML, removing stop words, and lemmatizing with spaCy, then choose the number of topics K by measuring c_v coherence across K values from 5 to 70; K=55 scores highest (0.6318) and is adopted. Each of the 55 topics is manually labeled using open card sorting, and the labels are organized hierarchically into two categories and nine subcategories. To measure change over time, the authors assign each post to its dominant topic and compute monthly absolute and relative impact metrics for the categories.","core_discovery":"The paper's central claim is that CS educators are not confined to technical discussion: across the Stack Exchange network they participate in a broad and evolving topic space, with IT topics (64.88%) dominating but non-IT topics (35.12%) forming a substantial and growing share. Using LDA topic modeling on a filtered corpus of 397,061 questions and 351,023 accepted answers from 12,949 CS Educators users, the authors identify 55 topics, manually label them, and organize them into two categories and nine subcategories. The largest IT subcategory, Programming and Software Development, accounts for about 23% of posts; the largest non-IT subcategory, Mathematics, accounts for about 8.22%. The temporal analysis shows IT leading absolute volume throughout 2008-2024, while non-IT topics gained relative ground after 2014, with steady growth from 2013 to 2018.","pith_inferences":["Beyond the paper: the accepted-answer filter probably excludes questions that never got resolved, often the hardest or most niche ones, so the reported 64.88/35.12 split may understate how much of CS educators' discussion is non-IT or emerging.","Beyond the paper: the 2013-2018 non-IT growth could partly be a platform-wide trend; comparing CS educators against a matched sample of other Stack Exchange users would isolate what is specific to educators.","Beyond the paper: the substantial mathematics, physics, and humanities activity suggests CS educators use the network for their own subject-matter learning, not only for teaching advice; a follow-up study could test whether such cross-disciplinary posting correlates with curriculum or course-design choices.","Beyond the paper: a direct extension would be an inter-rater reliability study of the manual labels; quantifying agreement on a sample would show how stable the 55-topic taxonomy is."],"forward_implications":["If the taxonomy holds, the nine-subcategory hierarchy gives educators, trainers, and platform designers a ready-made map of where CS educators actually spend their attention.","If the relative-impact trend continues, non-technical topics, especially mathematics, humanities, and lifestyle, will claim a growing share of CS educators' discussion, so professional development that ignores them will miss a real part of the job.","The 55-topic hierarchy can be reused as a baseline for comparing other expert communities, such as math educators or engineering educators, on the same network.","Because the absolute impact of IT topics remained highest throughout 2008-2024, any support system for CS educators must keep technical topics as the core while expanding to interdisciplinary ones."],"supporting_citations":[{"why":"Supplies the latent Dirichlet allocation model used to identify the 55 discussion topics.","marker":"[11]"},{"why":"Provides the MALLET implementation of LDA used to train the topic model.","marker":"[20]"},{"why":"Gives the coherence-based method for selecting the optimal number of topics (K=55).","marker":"[4]"},{"why":"Defines the c_v coherence metric used to assess topic quality and choose K.","marker":"[23]"},{"why":"Documents the Stack Exchange network and its 169 English-language sites that define the study's scope.","marker":"[26]"},{"why":"Supplies the data dump of posts, users, and tags used as the primary data source.","marker":"[27]"},{"why":"Provides the API used to retrieve account IDs that link one user's posts across sites.","marker":"[25]"},{"why":"Supplies the open card sorting method used for manual topic labeling.","marker":"[18]"},{"why":"Establishes the precedent of analyzing topics and trends in Stack Overflow posts that this study extends to CS educators.","marker":"[8]"},{"why":"Precedent for large-scale topic analysis and for the practice of analyzing only accepted answers.","marker":"[6]"}],"fun_headline_variants":["One in three CS educator posts isn't about IT","CS educators talk code, but a third is non-tech","Beyond programming: CS educators also tackle math and humanities","Study: CS educators' interests span tech and beyond","Math and humanities gain ground in CS educator discussions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The findings rest on the assumption that posts by users of the CS Educators site, filtered to questions that received an accepted answer, fairly represent what CS educators talk about across the network; if unanswered questions or unaccepted answers cover different topics, the 64.88/35.12 split and the 2013-2018 growth trend would be biased.","fun_headline_variants_meta":{"raw":{"variants":["One in three CS educator posts isn't about IT","CS educators talk code, but a third is non-tech","Beyond programming: CS educators also tackle math and humanities","Study: CS educators' interests span tech and beyond","Math and humanities gain ground in CS educator discussions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1469,"prompt_tokens":930,"completion_tokens":539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":546,"tokens_out":539,"duration_ms":6049,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:10:25.821330+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of posts excluded by the accepted-answer filter, apply the same preprocessing and LDA model, and compare the resulting category shares against the reported 64.88% IT / 35.12% Non-IT split; a material shift would show the filter distorted the claimed distribution.","supporting_citations":[{"cited_title":"Blei, Andrew Y","cited_arxiv_id":null,"evidence_quote":"Supplies the latent Dirichlet allocation model used to identify the 55 discussion topics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MALLET implementation of LDA used to train the topic model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the coherence-based method for selecting the optimal number of topics (K=55)."},{"cited_title":"Röder, A","cited_arxiv_id":null,"evidence_quote":"Defines the c_v coherence metric used to assess topic quality and choose K."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the Stack Exchange network and its 169 English-language sites that define the study's scope."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the data dump of posts, users, and tags used as the primary data source."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the API used to retrieve account IDs that link one user's posts across sites."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the open card sorting method used for manual topic labeling."},{"cited_title":"Barua, S","cited_arxiv_id":null,"evidence_quote":"Establishes the precedent of analyzing topics and trends in Stack Overflow posts that this study extends to CS educators."},{"cited_title":"Bagherzadeh and R","cited_arxiv_id":null,"evidence_quote":"Precedent for large-scale topic analysis and for the practice of analyzing only accepted answers."}],"review_version":1}