{"id":"27c36f3c-11c6-414f-b528-b313f9d8ce18","arxiv_id":"2506.05522","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Across 1,807 Mastodon instances and 12 moderator interviews, community-level blocklists vary widely in purpose and transparency, and moderators balance openness, safety, and context when using them.","lead":"This paper maps how Mastodon moderators use community-level blocklists, shared lists of servers to block, and finds wide variation in their goals, criteria, and transparency. It combines a scan of over 1,800 instances with interviews of 12 moderators to suggest practical design improvements for moderation tools.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 20.1% public-sharing statistic may undercount blocklists distributed off-platform; if confirmed, the 'limited transparency' finding and design recommendations lose their empirical anchor.","rationale":"The reader's weakest_assumption was representativeness of the curated sample; I agree that is a genuine limitation, but the more sharply checkable risk is the operationalization of 'publicly share' in the paper's quantitative headline. The paper counts only About-page listings, while all five blocklists it analyzes are distributed off-platform; this is not a sampling issue but a construct-validity issue in the central 20.1% result. A re-analysis that includes known off-platform channels is cheap, decisive, and can be done without new interviews. I do not find derivation-level failures: Table 3 counts are internally consistent, the interview quotes support the three decision styles, the selection criteria in §3.2.2 are stated, and §6 contains candid limitations. The single-coder thematic coding noted by the reader remains an important secondary concern, but it is less directly testable from public data than the off-platform sharing count. Because the reader's CONDITIONAL verdict already anticipates addressable weaknesses, my concern reinforces rather than moves that verdict.","tokens_in":22736,"tokens_out":8578,"duration_ms":90705,"concrete_test":"Re-run the §3.2.1 census for the same 1,807 instances at the same snapshot date, but augment the About-page scrape with three additional public channels: (a) domains listed in the five shared blocklists studied in §3.2.2 and their Git/CSV distribution points, (b) FediBlockHole/FediCheck subscription feeds associated with each instance's admin account, and (c) public #FediBlock posts or announced blocklist URLs from each instance's official account during the study period. Compute the revised share of instances with any public blocklist signal. If the revised percentage exceeds roughly 30% (a >50% relative increase over 20.1%), the headline statistic is an undercount and the 'limited transparency' conclusion needs correction or a narrower wording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical finding that only 364 of 1,807 instances (20.1%) publicly share blocklists (§4.1.1) is operationalized narrowly in §3.2.1: data were scraped only from each instance's publicly accessible About page. But the paper's own five shared blocklists (§3.2.2) are distributed through external channels—public CSVs, Git/Codeberg repositories, and synchronization tools such as FediBlockHole and FediCheck—rather than only through Mastodon About pages. If any substantial number of the 1,807 instances publish, subscribe to, or link their blocklists through those channels without rendering them on the About page, the 20.1% figure undercounts actual public sharing. That statistic is the quantitative anchor for the conclusion that 'explicit transparency in moderation remains limited' and for the design motivation around transparency-focused features (§5.3.2). This is a measurement-validity concern, not a disagreement with consensus: it is checkable and, if it lands, changes a core descriptive result. The paper's §6 limitations acknowledge selection skew but do not acknowledge the About-page-only measurement constraint.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies community-level blocklists in Mastodon, a decentralized social media platform. It combines a content analysis of the blocklist landscape (based on scraping 1,807 Mastodon instances with at least 10 active users, a curated selection of five shared blocklists, and four blocklist-related tools) with semi-structured interviews of 12 Mastodon moderators from seven countries. The central claims are that only 20.1% of instances publicly share blocklists; that shared blocklists vary in purpose, inclusion criteria, and transparency; that moderators balance three decision-making styles prioritizing openness, safety, and context; and that design improvements such as category filters, severity toggles, comment receipts, and collaborative voting would support moderation. The paper also presents a lightweight demo system used as a design probe during interviews, and derives implications for transparency and collaborative moderation in decentralized systems.","tokens_in":22903,"tokens_out":3986,"duration_ms":42381,"significance":"If the findings hold, the paper provides one of the first systematic, mixed-method characterizations of community-level blocklists in decentralized social media, filling a gap between individual-level blocklist research and platform-level moderation studies. The work has concrete strengths: a multi-method design; a multi-country interview sample with direct quotes; comparative tables of five blocklists and four tools; and a working prototype used as a design probe, which grounds the design recommendations in concrete participant reactions. The proposed design features and the three-part framework of moderation approaches are actionable for future research and tool development. The main quantitative anchor, however, is vulnerable to measurement-validity concerns described below, and the single-coder coding process limits confidence in the qualitative categories.","major_comments":[{"comment":"The claim that only 364 of 1,807 instances (20.1%) publicly share blocklists is operationalized by scraping only each instance's About page. However, the shared blocklists analyzed in §3.2.2 are explicitly distributed through external channels — public CSVs, Git/Codeberg repositories, and synchronization tools such as FediBlockHole and FediCheck — rather than through About pages alone. If any substantial number of the 1,807 instances publish, subscribe to, or link their blocklists through such channels without rendering them on the About page, the 20.1% figure undercounts actual public sharing. This statistic is load-bearing for the conclusion that 'explicit transparency in moderation remains limited' and for the design recommendations in §5.3.2. Please either broaden the measurement to include external blocklist channels (e.g., querying domain-block API endpoints, checking instance metadata, or searching public repositories) or reframe the claim as 'About-page-visible transparency,' and explicitly acknowledge this constraint in the limitations section, which currently only discusses selection skew of curated resources.","section":"§3.2.1 and §4.1.1"},{"comment":"The content analysis of blocklists and tools and the thematic analysis of interviews were both conducted by the first author alone, with no inter-rater reliability or second-coder agreement reported. The Tables 4 and 5 categories (Purpose, Criteria, Transparency, Distribution, etc.) and the three moderation styles reported in §4.2 are central to the paper's findings; a single-coder process without an agreement check leaves the reliability of these categorizations unestablished. Please report a second coder on a subset of the data with agreement metrics (e.g., Cohen's kappa), or explicitly justify and contextualize the single-coder approach with reference to established qualitative methods, and describe how the iterative author feedback changed the coding.","section":"§3.2.2 and §3.3.4"},{"comment":"The scraping pipeline is underspecified. The paper states only that 'automated scraping' with Octoparse was used to extract blocked domains and reasons from About pages, with no details on extraction rules, handling of dynamic or paginated content, validation or de-duplication of blocked domains, or how 'publicly share' was judged (e.g., whether any visible block section counts). This undermines reproducibility and makes it difficult for readers to assess potential measurement error in the headline 20.1% statistic. Please provide a detailed data-collection protocol, including the exact scrape date, URL patterns, inclusion/exclusion criteria, and ideally release the anonymized dataset or scripts so that the statistic can be independently checked.","section":"§3.2.1"}],"minor_comments":[{"comment":"The author list in the header contains 'SOHYEON HW ANG' with a spurious space; this should be corrected to 'Hwang'.","section":"Author list"},{"comment":"Footnote 11 is truncated ('e.g., NSFW content would not work for a professionally-oriented instance') and appears to be an incomplete sentence; please complete or integrate it into the main text.","section":"Footnote 11"},{"comment":"Table 3a reports percentages for each instance-size band but omits the denominators for the Yes and No columns; adding the totals would help readers verify the percentages and understand the unequal group sizes.","section":"Table 3a"},{"comment":"The paper does not provide the interview protocol or the final codebook; including these in an appendix would strengthen transparency and allow other researchers to replicate or extend the thematic analysis.","section":"Appendix / reproducibility"},{"comment":"A number of references have incomplete bibliographic details (e.g., [1], [33], [63] lack full venue or publication data); please bring all entries into consistent style for the target venue.","section":"References"},{"comment":"The disclosure of LLM usage in Section 8 is explicit and appropriate, and the described scope of use is consistent with common editorial assistance.","section":"§8"}],"recommendation":"major_revision","confidential_remarks":"The measurement-validity concern about the 20.1% statistic is credible and should be raised with the authors: the paper's own selected shared blocklists are distributed off About pages, so the current operationalization likely undercounts public sharing on at least some instances. The single-coder coding issue is common in qualitative work but should be addressed with a reliability check or a more explicit methodological justification. If these points are resolved, the paper would be a solid contribution to CSCW/social-computing literature on decentralized moderation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the most complete mixed-method map of the Mastodon blocklist ecosystem I've seen, and the design agenda (category filters, severity toggles, comment receipts, collaborative voting) is concretely grounded in interviews. The headline 20.1% sharing statistic is probably a lower bound, so read the transparency claims with that in mind.\n\nWhat's actually new: a content analysis of 1,807 instances, five shared blocklists, four tools, and twelve moderator interviews. Prior work looked at admin labor and individual-level blocklists; this connects the ecosystem-level picture to moderator practice. The openness/safety/context framework is a fair summary of the interview data, and the proposed features come directly from what moderators said they need. Credit where due: the paper is honest about its limitations, including English-language bias and the small interview sample.\n\nSoft spots, in proportion. The 20.1% figure comes from scraping only each instance's About page. But the five shared blocklists the authors themselves analyze are distributed via CSVs, Git repositories, and sync tools. Instances can share or subscribe to blocklists without rendering them on the About page. So the count of 364/1,807 is an undercount, and the paper never acknowledges that measurement constraint. That's a real gap. Even if the true rate were double, the conclusion that explicit transparency is limited would still hold—moderators still report trouble finding trustworthy blocklist information—so this is a fixable measurement issue, not one that sinks the paper.\n\nAlso soft: the blocklist/tool coding and thematic analysis were done by the first author alone, with no inter-rater reliability reported. The scraping pipeline is underspecified. No data or coding artifacts are released. These are addressable, and none undercut the main qualitative findings, which are well supported by participant quotes.\n\nWho this is for: people working on fediverse moderation, trust and safety tooling, and decentralized governance. It would be a solid CSCW/ICWSM paper after revision.\n\nRecommendation: send it to peer review. It deserves referee time; the reviewers will ask for the measurement caveat and coding transparency, but the core empirical contribution is real.","headline":"The most complete map yet of Mastodon blocklists, with a real but fixable measurement caveat in the headline transparency statistic.","tokens_in":23503,"tokens_out":2889,"would_cite":true,"duration_ms":26595,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper studies community-level blocklists on Mastodon, showing that most instances keep them private and that moderators balance openness, safety, and context when blocking entire communities, and it argues for redesigning blocklist…","keywords":["content moderation","blocklists","Mastodon","Fediverse","decentralized social media","moderation tools","community governance","qualitative interviews"],"falsifier":"A census or large random sample of Mastodon instances that includes non-English and very small instances could settle the generalization question: if those instances show a substantially different public-sharing rate, or if interviews with their moderators do not reproduce the three decision styles and the same design needs, then the paper's framework and recommendations would not generalize.","tokens_in":22522,"feed_emoji":"🛡️","tokens_out":5022,"duration_ms":53815,"temperature":0.7,"pith_summary":"This paper studies community-level blocklists on Mastodon, the lists by which one server blocks an entire other server from interacting with it. It claims that blocklists are the key cross-community moderation tool in decentralized social media, yet they are rarely shared and poorly understood. Combining a content analysis of 1,807 Mastodon instances, five shared blocklists, and four management tools with twelve moderator interviews, the authors find wide variation in blocklist goals, criteria, and transparency, and identify three moderation styles that balance openness, safety, and context. The result matters because if the paper is right, blocklist tooling should be redesigned around category filters, severity toggles, comment receipts, and collaborative decision-making rather than simple import-export lists.","feed_headline":"Only 1 in 5 Mastodon instances share their blocklists","feed_subtitle":"Interviews with 12 moderators reveal three moderation styles and the features blocklist tools still lack.","key_machinery":"The central object is the community-level blocklist itself, defined as a list by which a Mastodon instance blocks another entire instance. The argument is carried by two analytic instruments: a coding scheme that characterizes blocklists and tools along dimensions of purpose, criteria, transparency, distribution, limitations, and additional content, and an interview-derived framework of three moderator decision styles (responsive/openness, proactive/safety, measured/context). A lightweight prototype interface with category filters, severity toggles, and comments served as a design probe to elicit user needs. Together these allow the paper to move from describing the ecosystem to proposing concrete design directions.","core_discovery":"The paper's central claim is that community-level blocklists in the Fediverse are not monolithic: among 1,807 instances with at least 10 active users, only 364 (20.1%) publicly share blocklists, and fewer than half of those give explicit reasons; the shared blocklists that do exist differ in whether they seek broad consensus or target severe actors, and in how much they disclose their sources. Interviews with twelve moderators show that blocklist use follows three decision styles — responsive and openness-prioritizing, proactive and safety-prioritizing, and measured and context-prioritizing — which moderators move between rather than choose once. On this basis the paper argues that current tools fail to support the collaborative, transparent, and flexible practices moderators want, and that features such as category filters, severity toggles, comment receipts, and collaborative voting would better match how moderation actually happens.","pith_inferences":["We infer that the reported 20.1% public-sharing rate likely overstates ecosystem transparency, because the sample excluded instances with fewer than 10 active users, and the paper's own size trend suggests the smallest instances share least.","We infer that the openness/safety/context triangle could generalize as an analytic lens to other federated or protocol-based platforms beyond Mastodon, though the paper only studies Mastodon and cautions against such generalization.","We infer that standardizing tags and category definitions across blocklists, with multilingual support, could be tested as a way to reduce false-positive blocking and cross-instance misunderstandings, building on the paper's finding that tools currently lack standardized categorization."],"forward_implications":["Blocklist management tools should add category filters and severity toggles so moderators can see what an entry is about and choose an action level per instance.","Comment receipts and public annotations should become standard, since moderators already use private comments to keep decisions justifiable, renewable, and reviewable.","Design should support collaborative workflows, including voting, suspect lists, shared receipt libraries, and blocklist subscription and merging across instances.","Shared blocklists should be framed as starter resources with documented sources and biases, not as definitive lists, because they vary in consensus-based versus severity-based criteria.","Moderator support should accommodate movement among responsive, proactive, and manual-review styles rather than assuming one fixed workflow."],"supporting_citations":[{"why":"Establishes the baseline of blocklists as individual-level harassment-mitigation tools that this paper extends to community-level instances.","marker":"[48]"},{"why":"Shows how shared collective blocklists can form protective counterpublics, the prior work this study's community-level framing builds on.","marker":"[28]"},{"why":"Documents the challenges decentralized Fediverse moderators face, providing the context that blocklists address.","marker":"[4]"},{"why":"Supplies the prior finding that Mastodon admins moderate by blocking problematic users and instances, motivating the focus on blocklists.","marker":"[99]"},{"why":"Raises the tensions between community protection and mutual aid in federated blocklists, which the interviews echo.","marker":"[73]"},{"why":"Demonstrates transparency and dynamics problems in open-source blocklisting, informing the paper's transparency analysis.","marker":"[25]"},{"why":"Introduces the digital covenant governance idea that underpins the paper's consensus-building design recommendations.","marker":"[100]"},{"why":"Provides the standardized glossary of trust and safety terms used by tools like FediCheck and by the paper's category analysis.","marker":"[19]"}],"fun_headline_variants":["Only 1 in 5 Mastodon instances publish blocklists","Mastodon moderators shift among three blocklist styles","Blocklist tools miss features moderators request","Most Mastodon blocklists stay private with unclear criteria","Three moderation styles emerge from Mastodon interviews"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The findings rest on the assumption that the twelve moderators and the five curated blocklists and four tools represent the broader Mastodon ecosystem, even though the selection skews toward English-language, publicly visible, widely referenced resources and instances with at least ten active users.","fun_headline_variants_meta":{"raw":{"variants":["Only 1 in 5 Mastodon instances publish blocklists","Mastodon moderators shift among three blocklist styles","Blocklist tools miss features moderators request","Most Mastodon blocklists stay private with unclear criteria","Three moderation styles emerge from Mastodon interviews"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1617,"prompt_tokens":944,"completion_tokens":673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":598}},"tokens_in":560,"tokens_out":673,"duration_ms":7265,"temperature":1.0,"reasoning_tokens":598,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:19:30.301474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A census or large random sample of Mastodon instances that includes non-English and very small instances could settle the generalization question: if those instances show a substantially different public-sharing rate, or if interviews with their moderators do not reproduce the three decision styles and the same design needs, then the paper's framework and recommendations would not generalize.","supporting_citations":[{"cited_title":"Stuart Geiger","cited_arxiv_id":null,"evidence_quote":"Shows how shared collective blocklists can form protective counterpublics, the prior work this study's community-level framing builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prior finding that Mastodon admins moderate by blocking problematic users and instances, motivating the focus on blocklists."},{"cited_title":"A Blocklist is a Boundary","cited_arxiv_id":null,"evidence_quote":"Raises the tensions between community protection and mutual aid in federated blocklists, which the interviews echo."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the digital covenant governance idea that underpins the paper's consensus-building design recommendations."}],"review_version":1}