{"id":"ba15bce1-8358-4ca5-a561-e423aef9588e","arxiv_id":"2505.22702","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Bluesky mentions of scientific papers increased sharply after November 2024, and for two library science journals Bluesky user mentions overtook X, though X remains dominant overall.","lead":"This paper measures how often published research is discussed on X and on Bluesky before and after the November 2024 US election, using mentions from 14,497 papers in seven journals. It finds a sharp jump in Bluesky discussion after the election, and in two smaller journals Bluesky overtakes X in user mentions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-election 'Bluesky surpasses X' rests on tiny, skewed samples; QSS/JASIST reversal may be driven by a few papers and is not statistically supported.","rationale":"The reader's weakest assumption correctly identifies small, heterogeneous counts as the load-bearing weakness: QSS has only 86 papers total, the post-election subsample is smaller, and aggregate percentages can be dominated by a few highly-discussed papers. My stress test agrees with that diagnosis and adds a second, reinforcing fragility visible in Table 1: the two Bluesky measurement sources disagree enormously for QSS (0.60 vs 12.44 average accounts), meaning Figure 2's reversal depends on which source is used. This makes the 'Bluesky surpasses X' claim even less secure than the small-sample concern alone. However, the paper's broader descriptive finding—a rise in Bluesky activity after November 2024—is supported by the data and is not threatened by the critique. The authors are transparent about much of their methodology, share code, and acknowledge the exploratory nature of the work. A conditional acceptance that requires robustness checks, confidence intervals, and softer language about 'surpassing' matches the evidence. Since the reader already recommended CONDITIONAL, my analysis does not change the verdict.","tokens_in":4762,"tokens_out":3158,"duration_ms":37164,"concrete_test":"Recompute Figure 2 from the shared GitHub data restricted to papers published November 2024–March 2025, using per-paper unique-user counts. Report the full distribution, the Bluesky share with 95% bootstrap confidence intervals, and the share after removing papers above the 95th percentile of total mentions. Also rerun the same figure using the ad-hoc Bluesky counts instead of the Altmetric counts. If the QSS/JASIST majority disappears, the confidence intervals overlap 50%, or the result depends on the Bluesky source, the 'surpasses X' claim should be downgraded to a possible but unverified pattern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that after November 2024 Bluesky matches or exceeds X in unique-user mentions, with JASIST at 65% and QSS at 56%—depends entirely on the post-election subset in Figure 2. That subset is very small: QSS has only 86 papers in the full 15-month window, so the 5-month post-election window likely contains only a few dozen papers, and JASIST has 169. The paper reports only aggregate shares, with no per-paper distribution, confidence intervals, or stability test. User-mention data are highly right-skewed, so a handful of high-discussion papers can dominate the total and flip the majority. Table 1 itself illustrates the fragility: for QSS, the two Bluesky sources disagree by roughly a factor of 20 in average accounts (0.60 ad-hoc vs 12.44 Altmetric), and the claimed reversal is not visible in the ad-hoc series. Without a robustness check that excludes or down-weights outliers, the reader cannot tell whether 'Bluesky surpasses X' is a population-level shift or an artifact of a few papers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This brief communication examines whether researchers are shifting from X (Twitter) to Bluesky after the November 2024 US presidential election. The authors analyze 14,497 publications from three multidisciplinary journals (Nature, Science, PNAS) and four Library and Information Science journals (JASIST, Journal of Informetrics, Scientometrics, QSS) over January 2024–March 2025. They use two complementary data sources: a custom ad-hoc Bluesky API search and Altmetric's user counts for both X and Bluesky. The paper reports that Bluesky is a minority platform over the whole period, but that its share of unique-user mentions rises sharply after November 2024, surpassing X in JASIST (65%) and QSS (56%). The authors interpret this as evidence that the altmetric hegemony of X may have come to an end.","tokens_in":4909,"tokens_out":2252,"duration_ms":25725,"significance":"If robust, the findings would be valuable for the altmetrics and scholarly communication communities: they provide an early, transparently documented observation of a platform shift, with open code and explicit coverage rates. The paper also usefully highlights discrepancies between data sources (ad-hoc vs. Altmetric) that will matter to anyone building on altmetric indicators. However, the central comparative claim—Bluesky surpassing X in specific journals after the election—rests on small and potentially skewed subsamples, and the paper does not provide the statistical or robustness evidence needed to support that claim as stated. The paper is a useful exploratory descriptive study, but its headline conclusion needs stronger support before it can be accepted as a definitive finding.","major_comments":[{"comment":"The claim that Bluesky surpasses X in JASIST (65%) and QSS (56%) in the post-election period is based on very small subsamples. QSS has only 86 papers in the entire 15-month window, so the November 2024–March 2025 subset likely contains only a couple of dozen papers; JASIST has 169 papers total. User-mention distributions are highly right-skewed, so a handful of high-discussion papers could dominate the aggregate shares. The paper reports only aggregate percentages, with no per-paper distribution, confidence intervals, significance tests, or robustness checks. Please report the underlying counts (number of papers and mentions) for each bar in Figure 2, and add a sensitivity analysis that excludes or down-weights outlier papers, or provide bootstrap confidence intervals for the reported shares.","section":"§3, Figure 2"},{"comment":"For QSS, the two Bluesky sources disagree dramatically: the ad-hoc method reports an average of 0.60 accounts per paper, while Altmetric reports 12.44—a factor of roughly 20. Since the post-election reversal in Figure 2 is based on Altmetric user counts, the choice of data source is load-bearing for the paper's central claim. The manuscript does not adequately explain this discrepancy or examine which source is more reliable for the LIS journals. Please add a discussion of the source discrepancy and show how Figure 2 changes if the ad-hoc Bluesky counts are used instead of Altmetric counts.","section":"Table 1"},{"comment":"The ad-hoc method counts total mentions from the Bluesky API, while Altmetric provides counts of unique users. Table 1 reports 'Avg. Accounts' for both sources without clearly distinguishing total mentions (S1) from unique users (S2). Because the X comparison is based on unique users, the ad-hoc column is not directly comparable, and the juxtaposition may mislead readers about coverage and engagement. Please clarify the metric for each column and, where possible, use the same metric for all comparisons.","section":"§2, Methodology"},{"comment":"The text states that Nature mentions rose 'from 440 per week to 1,412 per week by mid-November,' but Figure 1's y-axis and aggregation are not clearly labeled (mentions per week? per month? raw counts?). Please specify the unit and aggregation window in the figure caption, and state whether the quoted numbers are weekly averages or raw weekly counts.","section":"§3, Figure 1"}],"minor_comments":[{"comment":"The abstract says the findings mark 'a significant change' and the discussion says the 'evidence ... is clear,' but no statistical significance tests are reported anywhere in the paper. Please qualify these statements as descriptive or add appropriate tests.","section":"Abstract"},{"comment":"Footnote 1 notes that Altmetric only captures posts with a direct link to the publication, which may underrepresent Bluesky or X activity. This limitation is load-bearing for the comparison and should be discussed in the main text or limitations, not only in a footnote.","section":"Footnote 1"},{"comment":"The discussion speculates about why Altmetric started tracking Bluesky (e.g., 'increasing opacity of X'). This speculation is not supported by data in the paper and could be labeled as conjecture or removed.","section":"§4, Discussion"},{"comment":"The reference list is missing a citation for the GitHub repository mentioned in §2; please add the repository URL as a footnote or reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a scientometrics or altmetrics venue, and the descriptive trends are likely of interest. However, the central claim that Bluesky surpasses X in two LIS journals is not supported with the current level of statistical or robustness evidence, and the large source discrepancy for QSS in Table 1 undermines confidence in the post-election reversal. These issues are fixable within the manuscript's scope, so major revision rather than rejection is appropriate. I would also encourage the authors to make the underlying datasets and analysis scripts fully available, as they have already done for the data retrieval script, so that reviewers can verify the small-sample results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core observation is real and useful: after November 2024, Bluesky's share of user mentions for these seven journals clearly rises, and for two US-based LIS journals it overtakes X. That is a new empirical timestamp for the altmetrics literature, and the paper is honest about its exploratory nature. Credit is due for the two-source retrieval approach (ad-hoc API plus Altmetric), the transparent reporting of coverage rates and averages, and the posted code on GitHub. The discrepancies between the two Bluesky sources—QSS shows 0.60 average accounts ad-hoc versus 12.44 via Altmetric—are not hidden; that is more than many altmetrics papers do.\n\nThe soft spots are real but not fatal. The post-election 'reversal' in JASIST and QSS is computed from very small pools: QSS has 86 papers in the full period, so the November–March subset is small, and a few high-discussion papers could flip the percentages. No confidence intervals, no per-paper distribution, no outlier check. The mismatch between the two Bluesky sources makes the aggregate shares even more fragile. The stress-test note is right about this. However, the paper's own framing mostly stays cautious: the word 'exploratory' appears, and the conclusion says 'may have come to an end,' not 'has ended.' The larger generalization about 'the altmetric hegemony of X' is overreach, and I would push the authors to soften that line.\n\nMethodologically, this is not a novel approach—it is standard altmetrics description applied to a new platform—but that is not a flaw. The citation pattern is clean; the self-citation to their own prior community-detection work is incidental. The central descriptive trend is directly from API data and the field is moving fast enough that this kind of snapshot deserves a record.\n\nWho is this for? Researchers working on altmetrics, science communication, and scholarly social-media migration. They will get value from the descriptive numbers and the timing. For peer review, I would send it out: it is a short communication, clearly written, and the weaknesses are addressable with minimal additional work—confidence intervals, per-paper breakdowns, or a sentence acknowledging that the reversal is not yet statistically robust. A serious referee could push it to a solid contribution.","headline":"Timely empirical snapshot of Bluesky's post-election rise in altmetrics, but the 'surpasses X' claim rests on very small samples and needs robustness work.","tokens_in":5425,"tokens_out":1464,"would_cite":true,"duration_ms":18360,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper finds that after the November 2024 US presidential election, Bluesky's share of unique users mentioning scientific papers rose sharply and exceeded X's in two Library and Information Science journals.","keywords":["Bluesky","X","Twitter","Social Media","Scientific Communication","Altmetrics","User migration","Scholarly communication"],"falsifier":"Recompute the post-election platform shares for JASIST and QSS after removing the five most-mentioned papers in each journal; if Bluesky's share no longer exceeds X's, the headline reversal is an artifact of small counts. Extending the same analysis through a later six-month window would also show whether the reversal persists or was a short-lived spike.","tokens_in":4547,"feed_emoji":"🦋","tokens_out":10408,"duration_ms":110477,"temperature":0.7,"pith_summary":"This paper tracks mentions of 14,497 publications from seven journals on X and Bluesky between January 2024 and March 2025. It tries to establish that the November 2024 United States presidential election marked a turning point in where scientists discuss published research: Bluesky's share of unique users who mention papers rose sharply in the following months and exceeded X's in two library and information science journals. The authors interpret the shift as diversification of conversation across two similar platforms rather than as a simple abandonment of X, and they describe it as the first clear alternative to X in the altmetrics landscape. If the finding holds, it matters because indicators that count only X would increasingly underrepresent actual scholarly discussion.","feed_headline":"Post-election surge lifts Bluesky above X in two journals","feed_subtitle":"From November 2024 to March 2025, Bluesky's unique users surpassed X's in JASIST and QSS.","key_machinery":"The analysis rests on comparing counts of unique social-media accounts that mention a paper, drawn from two retrieval paths. The first path is a custom script that resolves each publication's DOI, collects alternative URLs from an open bibliographic index, and queries the Bluesky API directly; the second uses a commercial altmetric service's API to obtain unique-user counts for both X and Bluesky. The temporal split at November 2024 is the key comparison device, contrasting the full period, in which X dominates, with the post-election months, in which Bluesky's share grows sharply. The journal selection acts as a second control, separating broad multidisciplinary titles from field-specific library and information science journals.","core_discovery":"The paper's central claim is that, measured by the number of unique accounts mentioning a publication, scholarly conversation is no longer concentrated on X. Over the whole period X still dominates, holding more than 80 percent of mentions in multidisciplinary journals and more in most library and information science journals, but from November 2024 through March 2025 the distribution becomes much more even. In JASIST, 65 percent of unique user mentions go to Bluesky, and in Quantitative Science Studies the figure is 56 percent, so Bluesky surpasses X in both. The authors connect the timing to political events and to the resulting wave of user migration, but they stress that the durability of the shift is still unknown.","pith_inferences":["The same data could test whether the post-election surge is driven by a small set of politically salient papers; if so, the aggregate reversal might not generalize across all of science.","The large gap between the two Bluesky counting methods implies that 'a Bluesky mention' is not yet a standardized unit, so comparisons between future studies will be fragile until a shared definition is adopted.","If Bluesky and X host different communities, combining their mention counts into a single indicator would mix signals; a natural extension is to compare the topics and author types active on each platform.","If the shift persists, evaluation exercises that rely on past X-based altmetrics would be biased toward disciplines and countries with slower Bluesky adoption."],"forward_implications":["Altmetric indicators that count only X will miss a growing share of the online discussion of papers, making X-only scores increasingly incomplete after 2024.","Field and country of the journal shape platform adoption, so discipline-level normalization is needed before comparing mentions across fields.","For large multidisciplinary journals, Bluesky's coverage can approach X's, meaning the platform is no longer a niche source for high-visibility research.","Future altmetric research should treat X and Bluesky as complementary channels with distinct user communities rather than as substitutes."],"supporting_citations":[{"why":"Documents the date and decision to begin tracking Bluesky, which is why Bluesky user counts exist in the altmetric data used here.","marker":"(Altmetric, 2024)"},{"why":"Provides external evidence of the post-election surge in Bluesky users that the paper ties to the observed mention surge.","marker":"(Holterman, 2024)"},{"why":"Supplies the push-pull migration model the paper uses to interpret the platform shift as coexistence rather than abandonment.","marker":"(Hou & Shiau, 2019)"},{"why":"Raises the question of what altmetric mentions mean, the interpretive frame the paper extends to a two-platform landscape.","marker":"(Robinson-Garcia et al., 2017)"},{"why":"Contextualizes user migration across social platforms as a recurring phenomenon affecting scientific communities.","marker":"(Jeong et al., 2023)"}],"fun_headline_variants":["Bluesky overtakes X in two LIS journals after US election","Post-election Bluesky surpasses X in JASIST and QSS","In two LIS journals, Bluesky beats X post-election","X loses ground to Bluesky in scholarly mentions post-election"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The post-election comparison treats small and uneven journal samples—QSS has only 86 papers in the full period and fewer in the short window—as if the resulting percentages were stable, although a few highly mentioned papers could drive the apparent Bluesky lead.","fun_headline_variants_meta":{"raw":{"variants":["Bluesky overtakes X in two LIS journals after US election","Post-election Bluesky surpasses X in JASIST and QSS","In two LIS journals, Bluesky beats X post-election","X loses ground to Bluesky in scholarly mentions post-election"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001348,"raw_usage":{"total_tokens":5433,"prompt_tokens":862,"completion_tokens":4571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":4497}},"tokens_in":478,"tokens_out":4571,"duration_ms":32695,"temperature":1.0,"reasoning_tokens":4497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:01:48.824732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the post-election platform shares for JASIST and QSS after removing the five most-mentioned papers in each journal; if Bluesky's share no longer exceeds X's, the headline reversal is an artifact of small counts. Extending the same analysis through a later six-month window would also show whether the reversal persists or was a short-lived spike.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the date and decision to begin tracking Bluesky, which is why Bluesky user counts exist in the altmetric data used here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides external evidence of the post-election surge in Bluesky users that the paper ties to the observed mention surge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Raises the question of what altmetric mentions mean, the interpretive frame the paper extends to a two-platform landscape."},{"cited_title":"User Migration across Multiple Social Media Platforms","cited_arxiv_id":"2309.12613","evidence_quote":"Contextualizes user migration across social platforms as a recurring phenomenon affecting scientific communities."}],"review_version":1}