Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Decoding the Rule Book: Extracting Hidden Moderation Criteria from Reddit Communities

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read Moderation criteria in Reddit communities can be extracted as explicit per-phrase removal scores using an interpretable transformer, and these scores match neural classifiers while revealing how similar norms are enforced differently.

desk verdict Useful application of PAT to Reddit moderation, but the CriteriaMatrix term scores are not validated and the central claim overstates what they establish. read the letter →

arxiv 2509.02926 v1 pith:M3FKYAYE submitted 2025-09-03 cs.CL

classification cs.CL
keywords contentmoderationRedditinterpretabilitylexicalscoringPartialAttentionTransformernormdiscoverycommunity-specificrulestoxicity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to turn the hidden, unwritten rules of Reddit moderators into an explicit, inspectable table: for 60 subreddits, it scores tens of thousands of short phrases by how strongly their presence predicts that a comment will be removed. The bet is that even though moderation decisions are contextual, they leave a reliable lexical footprint—phrases like "should I use" in fantasyfootball or "San Francisco" in Games carry community-specific removal risk. If this is right, communities no longer need to be treated as opaque rule boxes; their enforcement can be read, compared, and audited at the level of individual words and n-grams. The authors show the phrase-level scores reach accuracy close to a full-text fine-tuned BERT and better than ChatGPT on community-specific rules, then use the scores to uncover patterns such as blanket intolerance of any mention of "mod" and a spectrum of personal-attack subtypes tolerated differently across communities.

What carries the argument

Partial Attention Transformer (PAT): an interpretable text classifier that, for each input, partitions the comment into two spans, encodes each with BERT, sums the span scores, and trains on the full-comment removal label. During inference, a phrase alone is fed in and its softmax output becomes the term's moderation score. This lets a single model that never sees phrase-level labels produce calibrated phrase-level probabilities, which are assembled into the CriteriaMatrix—the shared vocabulary scored for each subreddit that carries the whole comparison.

What would settle it

Audit the criteria matrix directly: ask moderators of the studied subreddits to judge whether the top-scoring phrases are genuine violations, and test the PAT scores on later, out-of-time removal data against a context-aware full-text model. If moderators mostly reject the phrases, or if phrase scores lose most of their predictive power while the full-text model does not, the matrix is a description of the classifier rather than of community criteria.

Watch

Extended reading notes

Core claim

The central claim is that moderation criteria are "score tables of lexical expressions associated with content removal" and these tables can be produced automatically from historical deletion data. Using Partial Attention Transformer (PAT), a BERT-based model trained to predict removal from a comment while forcing its prediction to decompose into scores for text spans, the authors assign each candidate phrase in a shared vocabulary a 0-1 probability that a comment containing it gets moderated in a given subreddit. The resulting CriteriaMatrix reproduces neural model performance (PAT Bipartite F1 0.80 vs BERT FT 0.81) and yields interpretable, verifiable terms. The paper argues the matrix exp

Load-bearing premise

The load-bearing premise is that moderator decisions leave a reliable surface-lexical fingerprint: a phrase's removal score, learned from a balanced 50/50 sample, faithfully represents the community's real enforcement criteria rather than the model's own biases.

Editorial extensions

If this is right

  • Moderators and platform analysts can audit enforcement by inspecting the top-scoring phrases rather than reading policy text.
  • Enforcement differences among communities sharing the same stated rule can be quantified, enabling moderation calibration to local norms.
  • The same PAT pipeline can be applied to any classification task with subdomains that may apply different criteria, not just Reddit.
  • Supervised moderation models can be screened for unintended learned biases, such as flagging neutral "mod" mentions, before deployment.
  • Because PAT Window gets near-BERT performance while using only short spans, term-level criteria are usable as lightweight screening rules in high-volume settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the criteria matrix may be as much a map of the classifier's decision boundary as of moderator intent; the authors' own spurious-correlation caveat suggests validating term scores against moderator-labeled removal reasons or prospective moderation decisions.
  • The 50/50 balanced training sample likely inflates the apparent importance of rare phrases and could alter cross-community rankings; an inference is that sampling at the natural ~5% removal rate would yield a different, perhaps more policy-faithful, matrix.
  • The clustering of personal attacks suggests a testable extension: if communities tolerate different attack subtypes, toxicity classifiers tuned per subtype rather than one global threshold should improve both precision and user satisfaction.
  • Since the data come from 2016–2017, patterns like high Trump scores or location terms in r/Games may be time-bound; applying the method to current data would reveal drift in enforcement criteria.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes to extract explicit moderation criteria from historical Reddit moderation decisions by training a Partial Attention Transformer (PAT) per subreddit and using the model to assign 0-1 probabilities to individual vocabulary terms. These scores form a 'CriteriaMatrix' that is then used to compare community-specific enforcement patterns, to study moderator-related terms, and to cluster personal-attack language into subcategories. The authors report that PAT achieves classification performance 'comparable' to ChatGPT and 'effectively replicate[s] the performance of neural moderation models,' and they present qualitative analyses of r/fantasyfootball, r/Games, the term 'mod,' and nine personal-attack clusters as evidence of hidden, community-specific moderation criteria.

Significance. If the central claim were fully supported, the paper would make a useful contribution: a globally interpretable, lexical representation of moderation behavior that can be compared across communities, with potential practical value for moderators and for auditing automated moderation systems. The paper also has genuine strengths: it uses a widely studied moderation dataset, evaluates PAT in held-out classification tasks, reports concrete F1/AUC numbers, and explicitly acknowledges limitations such as spurious correlations and missing conversational context. However, the empirical support for the central claim is incomplete. The evaluation that is most relevant to the CriteriaMatrix—PAT's term-level inference mode—is not directly validated, and the headline performance claim is overstated relative to the paper's own Table 3. The qualitative findings are interesting but currently rest on post-hoc selection and manual inspection rather than on a validated measurement instrument.

major comments (4)
  1. [§3.5, Eq. (3), §5.1-5.3] The standalone term scores used to build CriteriaMatrix are computed by applying Eq. (3) to an isolated term, but PAT is trained only on the bipartite aggregation in Eq. (2). The model never sees single sequences during training, so these term-level inputs are out-of-distribution. The paper itself acknowledges in §3.5 that ρ1 may be inaccurate for very short sequences, yet PAT (Window)—the inference mode closest to term-level scoring—is evaluated by averaging over three-token windows in full comments, not by evaluating the actual standalone scores used in CriteriaMatrix. Consequently, the core interpretable output is not validated as a faithful proxy for moderators' criteria. Every downstream analysis in Sections 5.1-5.3 inherits this gap. The authors should provide direct evidence that term-level PAT scores track held-out removal rates (e.g., compare scores with empirical removal propor
  2. [Abstract, §4, Table 3] The claim that extracted lexical patterns 'effectively replicate the performance of neural moderation models' is not supported by the reported numbers. In Table 3, subreddit-aware BERT (FT) achieves F1 0.81 / AUC 0.90, PAT (Bipartite) achieves 0.80 / 0.89, and PAT (Window) achieves 0.69 / 0.83. Since CriteriaMatrix is built from PAT's short-span scoring mode, PAT (Window) is the relevant comparison, not PAT (Bipartite). A 12-point F1 gap and a 7-point AUC gap relative to BERT (FT) is a meaningful degradation, not 'effectively' replication. The authors should either recalibrate the wording or, better, add an evaluation that directly measures the predictive accuracy of the extracted score tables (e.g., ranking comments by their maximum term score and measuring AUC on held-out moderation outcomes).
  3. [§5.1, Figure 3] The community-specific norm analysis is based on an explicitly post-hoc selection of two subreddits with the largest BERT-vs-ChatGPT F1 gaps, followed by manual inspection of terms. The r/Games interpretation—that game titles/company names are removed for 'not providing additional values' and that location names reflect strict off-topic enforcement—is illustrated with a small number of examples and no systematic verification. The authors themselves note that for r/fantasyfootball they cannot directly verify whether flagged phrases occurred in rule-violating threads. These findings are plausible hypotheses, but they do not establish that the CriteriaMatrix 'reveals' enforcement criteria. To support the central claim, the authors could formulate a small number of pre-registered or at least clearly separated hypotheses and test them on held-out data, or provide structured annotation of a ra
  4. [§5.3, Table 5, Figure 4] The claim that personal-attack clusters reveal 'underlying subcategories of the toxic speech classification' depends on a k-means clustering with k=100 chosen without stated justification, a self-reported Silhouette score of 0.15, and manual categorization into named clusters. A Silhouette score of 0.15 indicates weak clustering structure, and the cluster names in Table 8 are acknowledged to be illustrative rather than precise. Moreover, Figure 4 plots per-subreddit score differences for six unnamed subreddits without confidence intervals or statistical tests for the claimed community-specific tolerances. These results are suggestive but should be presented as exploratory, or the authors should supply quantitative support (e.g., evaluation of cluster stability across seeds, permutation tests for subreddit-by-cluster interactions).
minor comments (5)
  1. [§1, Introduction] There are several broken sentences and typos, e.g., 'PAT is further used to build analysis using PAT provides various insights' and 'We hypothesize that they can be meaningfully represented.' The paper would benefit from a careful copyedit.
  2. [§3.6] The vocabulary construction is underspecified: 'For each n values, we selected the top 10,000 most frequent n-gram terms, based on probability scoring from an off-the-shelf large language model, Llama-3' is unclear. What 'probability scoring' is used, and why is it needed if the n-grams are selected by frequency? Also, the 'high-score span extraction threshold' is not defined, making the pipeline difficult to reproduce.
  3. [Table 3] The header 'PAT (Bipratite)' contains a typo; should be 'PAT (Bipartite)'.
  4. [Table 8] Several sample terms appear to be subword tokens (e.g., 'youll', 'you don', 'you a', 'you are'), which weakens the readability of the cluster examples. Please clarify whether these are raw tokens or detokenized phrases.
  5. [§5.2] The synthetic-comment experiment uses 50 ChatGPT/Claude-generated comments and finds that 16 of 60 classifiers flag all 50 as needing moderation. The paper correctly lists possible explanations, but the base rate of 'mod' removal in real data is also high (over 80% in 34% of subreddits). It would strengthen the analysis to report the empirical removal rate for 'mod' comments conditional on the specific synthetic comment templates, or to compare with a set of control terms with similarly frequent mentions of other community roles.

Circularity Check

1 steps flagged · score 5.0 of 10

CriteriaMatrix term scores are the fitted outputs of a classifier trained on the same removal labels they are claimed to reveal; partial circularity, mitigated by held-out classification and one external validation.

  1. fitted input called prediction [Section 3.1, Section 3.6, Section 5.1]
    "We conceptualize the challenge of understanding community-specific moderation as a vocabulary scoring problem... our goal is to build a vocabulary V ... and community-specific score matrices S_i ... where each element s_j^i indicates the contribution of term v_j to moderation decisions in community i. ... Finally, we apply each PAT_i to score all terms in vocabulary, to get a score matrix M, where M_i,j indicates the score that PAT for subreddit i has assigned to a term j."

    The 'contribution of a term to moderation decisions' is operationally defined as the score assigned by PAT, a model trained on the very same moderation decisions (removal labels). The CriteriaMatrix is therefore, by construction, the fitted decision surface of the classifier, not an independently measured moderator criterion. When the paper interprets high-scoring terms as evidence of community norms (e.g., r/Games locations, r/fantasyfootball phrases), it is re-describing the model's learned associations. The paper does validate PAT on held-out labels and checks real 'mod' removal rates, but those do not establish that most term scores correspond to moderators' true intent. The paper itself concedes in Section 5.4 that 'supervised models may learn spurious correlations rather than moderat

full rationale

The paper's central representation, the CriteriaMatrix, is built by applying a trained PAT model to a vocabulary and recording its softmax scores. Since PAT is trained to predict whether a comment was removed, the scores are, by definition, the model's fitted function of the training labels. Presenting these scores as 'moderation criteria' is therefore a form of fitted-input-called-prediction: the extracted criteria are the classifier's own outputs, not an external ground truth about moderator intent. This is the main circular step. However, the circularity is partial rather than total. The paper evaluates PAT's classification performance on held-out data (Table 3), which shows the learned patterns are predictive of removal outcomes in a generalizable way. The 'blaming moderators' analysis (Section 5.2) also checks actual moderation rates of comments containing 'mod', providing an external anchor for one specific finding. Additionally, the paper's Limitations section acknowledges missing context, balanced sampling, and the risk of spurious correlations, which tempers the strength of the claims. The self-citation to prior PAT work (Kim et al., 2023) is not load-bearing in a circular way; the architecture is described and evaluated rather than assumed correct via citation. Overall, the central 'criteria discovery' claim reduces in large part to the fitted model's outputs, but independent predictive checks and one external validation keep it from being fully definitional. Score 5 reflects this partial circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the lexical-representation hypothesis, the assumption that removal labels are valid signals of moderator criteria, and the use of PAT's term-level scores as calibrated probabilities. No invented physical entities are introduced. Several analysis parameters (k=100, window size, vocabulary thresholds) are chosen ad hoc.

free parameters (4)
  • k-means cluster count k = 100
    Section 5.3: k=100 chosen and filtered by distance to centroid; no principled justification; silhouette 0.15 indicates weak structure.
  • n-gram vocabulary threshold = top 10,000 per n
    Section 3.6: top 10,000 most frequent n-grams selected based on Llama-3 probability scoring; arbitrary.
  • PAT (Window) window size = 3 tokens
    Section 3.5: three-token windows chosen for scoring; no sensitivity analysis.
  • High-score span extraction threshold = unspecified
    Section 3.6: 'spans of texts that received high scores' sampled from 1,000 comments; threshold not defined.
assumptions (4)
  • domain assumption Moderation decisions can be meaningfully represented by predictive lexical patterns.
    Section 3.1 states this as the explicit hypothesis underlying the entire CriteriaMatrix construction.
  • domain assumption Labels y indicate genuine moderator violations.
    Section 3.2 uses removed comments from Chandrasekharan et al. (2018) as ground truth; 24-hour delay and balanced sampling may distort this.
  • domain assumption PAT term-level output rho is a well-calibrated probability of moderation.
    Sections 3.4-3.5 treat softmax(PAT(term)) as a meaningful moderation score; no calibration analysis is provided.
  • domain assumption Llama-3 probability scoring selects natural n-grams.
    Section 3.6 uses Llama-3 to pick the top 10,000 most frequent n-grams; this injects a separate model's notion of naturalness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoding the Rule Book: Extracting Hidden Moderation Criteria from Reddit Communities." pith.science (2026). https://pith.science/paper/M3FKYAYE

@misc{pith2026250902926,
  author       = {Pith},
  title        = {Pith review of: Decoding the Rule Book: Extracting Hidden Moderation Criteria from Reddit Communities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M3FKYAYE}},
  note         = {Machine review of arXiv:2509.02926}
}
read the original abstract

Effective content moderation systems require explicit classification criteria, yet online communities like subreddits often operate with diverse, implicit standards. This work introduces a novel approach to identify and extract these implicit criteria from historical moderation data using an interpretable architecture. We represent moderation criteria as score tables of lexical expressions associated with content removal, enabling systematic comparison across different communities. Our experiments demonstrate that these extracted lexical patterns effectively replicate the performance of neural moderation models while providing transparent insights into decision-making processes. The resulting criteria matrix reveals significant variations in how seemingly shared norms are actually enforced, uncovering previously undocumented moderation patterns including community-specific tolerances for language, features for topical restrictions, and underlying subcategories of the toxic speech classification.

Figures

Figures reproduced from arXiv: 2509.02926 by the authors.

Figure 1
Figure 1. Overview of criteria discovery. For each subreddit, a text classifier (PAT) is trained from past moderation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. PAT training architecture. A comment text [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Representative terms with large score differ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Average moderation scores for personal attack [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 18 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Shamim Ahmed, M Shamim Kaiser, Mohammad Shahadat Hossain, and Karl Andersson. 2024. A comparative analysis of lime and shap interpreters with explainable ml-based diabetes predictions. IEEE Access

  4. [4]

    Nadia Burkart and Marco F Huber. 2021. A survey on the explainability of supervised machine learning. Journal of Artificial Intelligence Research, 70:245--317

  5. [5]

    Eshwar Chandrasekharan, Chaitrali Gandhi, Matthew Wortley Mustelier, and Eric Gilbert. 2019. Crossmod: A cross-community learning-based system to assist reddit moderators. Proceedings of the ACM on human-computer interaction, 3(CSCW):1--30

  6. [6]

    Eshwar Chandrasekharan, Mattia Samory, Shagun Jhaver, Hunter Charvat, Amy Bruckman, Cliff Lampe, Jacob Eisenstein, and Eric Gilbert. 2018. The internet's hidden rules: An empirical study of reddit norm violations at micro, meso, and macro scales. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1--25

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186

  8. [8]

    Casey Fiesler, Jialun Jiang, Joshua McCann, Kyle Frye, and Jed Brubaker. 2018. Reddit rules! characterizing an ecosystem of governance. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12

Show all 25 references
  1. [9]

    Agam Goyal, Charlotte Lambert, Yoshee Jain, and Eshwar Chandrasekharan. 2024. Uncovering the internet's hidden values: An empirical study of desirable behavior using highly-upvoted content on reddit. arXiv preprint arXiv:2410.13036

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

  3. [11]

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509

  4. [12]

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674

  5. [13]

    Youngwoo Kim, Razieh Rahimi, and James Allan. 2023. https://aclanthology.org/2023.findings-emnlp.456 Conditional natural language inference . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6833--6851, Singapore. Association for Computational Li...

  6. [14]

    Youngwoo Kim, Razieh Rahimi, and James Allan. 2024. Discovering biases in information retrieval models using relevance thesaurus as global explanation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19530--19547

  7. [15]

    Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. 2024. Llm-mod: Can large language models assist content moderation? In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1--8

  8. [16]

    positive reinforcement helps breed positive behavior

    Charlotte Lambert, Frederick Choi, and Eshwar Chandrasekharan. 2024. " positive reinforcement helps breed positive behavior": Moderator perspectives on encouraging desirable behavior. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2):1--33

  9. [17]

    Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30

  10. [18]

    Yair Neuman and Yochai Cohen. 2023. Ai for identifying social norm violation. Scientific Reports, 13(1):8103

  11. [19]

    Chan Young Park, Shuyue Stella Li, Hayoung Jung, Svitlana Volkova, Tanu Mitra, David Jurgens, and Yulia Tsvetkov. 2024. Valuescope: Unveiling implicit norms and values via return potential model of social interactions. In Findings of the Association for Computational Linguisti...

  12. [20]

    Chan Young Park, Julia Mendelsohn, Karthik Radhakrishnan, Kinjal Jain, Tushar Kanakagiri, David Jurgens, and Yulia Tsvetkov. 2021. Detecting community sensitive norm violations in online conversations. In Findings of the Association for Computational Linguistics: EMNLP 2021, p...

  13. [21]

    Richard Phillips, Kyu Hyun Chang, and Sorelle A Friedler. 2018. Interpretable active learning. In Conference on fairness, accountability and transparency, pages 49--61. PMLR

  14. [22]

    Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, and Yejin Choi. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.812 What makes it ok to set a fire? iterative self-distillation of contexts and rationales for disambiguating d...

  15. [23]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. https://doi.org/10.1145/2939672.2939778 "why should i trust you?": Explaining the predictions of any classifier . In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining...

  16. [24]

    Paul R \"o ttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. Hatecheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int...

  17. [25]

    Galen Weld, Amy X Zhang, and Tim Althoff. 2024. Making online communities ‘better’: a taxonomy of community values on reddit. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages 1611--1633

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.