pith. sign in

arxiv: 2606.07522 · v1 · pith:JWAOWORE · submitted 2026-04-19 · cs.CL · cs.LG· cs.SI

Community-Specific Slang and Entity Detection via Semantic Shift in Fine-Tuned Language Models

Julia Kruk , Sanchita Porwal , Amitrajit Bhattacharjee , Mansi Phute This is my paper

pith:JWAOWOREreviewed 2026-07-05 16:47 UTCmodel glm-5.2open to challenge →

classification cs.CL cs.LGcs.SI
keywords semantic shiftword embeddingsslang detectioncommunity-specific languagefine-tuningDistilRoBERTaRedditunsupervised lexicon discovery
0
0 comments X

The pith

Words that move most in a fine-tuned model reveal community slang

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes that community-specific slang, named entities, and shared folklore can be discovered without labels by measuring how much each word's embedding changes when a pretrained language model is fine-tuned on a community's text. The authors fine-tune DistilRoBERTa on three Reddit communities (r/Technology, r/Gaming, r/WorldofWarcraft), then compare each word's embedding in the base model versus the fine-tuned model using cosine similarity. Words with the lowest cosine similarity between the two models are interpreted as having undergone the greatest semantic shift, and the authors argue that these are precisely the community-specific terms. They show qualitatively that the bottom 10-percentile of cosine similarity values yields recognizable community slang and entities (e.g., game-specific lore terms for r/WorldofWarcraft), while the top 10-percentile yields generic English words. The method is entirely unsupervised: no slang labels are needed for training, only a community text corpus. The authors also observe that a model fine-tuned on r/Gaming performs reasonably on r/WorldofWarcraft masked-word predictions, suggesting cross-community transfer when communities are topically related.

Core claim

The central object is the cosine similarity between a word's embedding in a base pretrained model and the same word's embedding after community-specific fine-tuning. The paper claims that this single quantity — a measure of how far a word has moved in embedding space — functions as an unsupervised discriminator: community-specific slang, entities, and folklore concentrate at the low-similarity end (high semantic shift), while general-purpose vocabulary concentrates at the high-similarity end. The authors demonstrate this on three Reddit communities by pulling the bottom 10-percentile and showing that the retrieved words are community-specific, though evaluation is primarily qualitative.

What carries the argument

Cosine similarity between base-model and fine-tuned-model word embeddings as a proxy for semantic shift; DistilRoBERTa as the base masked-language model; community-specific fine-tuning on Reddit subreddit corpora; bottom-10-percentile thresholding to isolate candidate community-specific terms.

If this is right

  • If semantic shift magnitude reliably tracks community-specific vocabulary, any online community with sufficient text could be profiled for its slang and entities without manual annotation or labeled training data.
  • The method could be integrated into hate-speech detection systems to flag words whose community-specific meaning diverges from standard usage, reducing false positives on dialectal or in-group language.
  • Communities sharing similar high-shift vocabulary could be identified as having overlapping user bases or cultural exchange, enabling unsupervised mapping of community relationships.
  • The approach could generalize beyond Reddit to any domain-specific corpus — professional jargon, subcultural forums, or regional dialects — wherever a pretrained model can be fine-tuned on enough in-domain text.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method's validity hinges on whether semantic shift is driven by genuine community-specific meaning change rather than by confounds like word frequency, token rarity, or training noise. A word that is rare in the base model's pretraining data but common in the community corpus might shift dramatically in embedding space without being 'slang' per se — it could simply be a domain term the model i
  • A natural test would be to control for frequency: partition words into frequency bins and check whether the semantic-shift signal survives within each bin. If low-frequency words dominate the low-similarity tail regardless of community specificity, the method may be measuring frequency effects rather than meaning change.
  • The cross-community transfer observed between r/Gaming and r/WorldofWarcraft suggests that semantic shift could be used to construct a similarity graph over communities — communities whose high-shift words overlap substantially would be connected — but this would require a quantitative overlap metric the paper does not define.
  • The F1 scores reported (0.0–0.064) on the masked-language-modeling evaluation suggest the fine-tuned models are not actually good at predicting the masked slang tokens, which raises the question of whether the embedding-shift signal and the model's predictive ability are measuring different things — the model may move a word's embedding substantially without learning to produce it in context.

Load-bearing premise

The paper assumes that a large drop in cosine similarity between a word's embedding before and after community-specific fine-tuning is caused by the word having community-specific meaning, rather than by confounds such as word frequency, tokenization artifacts, or random training dynamics. No statistical test or frequency control is provided to separate these possibilities.

What would settle it

If words that are merely rare in the base model's pretraining data — but have no community-specific meaning — also show large embedding shifts after fine-tuning, then the low-cosine-similarity tail would be dominated by frequency effects rather than genuine slang, and the method would not distinguish community-specific vocabulary from general low-frequency vocabulary.

Figures

Figures reproduced from arXiv: 2606.07522 by Amitrajit Bhattacharjee, Julia Kruk, Mansi Phute, Sanchita Porwal.

Figure 1
Figure 1. Figure 1: Semantic Shift across Reddit communities. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Technology community [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: World of Warcraft 7.1 Limitations Due to computational limitations, we were only able to explore rel￾atively smaller communities. Communities with larger amounts of data, like (League of Legends) may have provided further validation to our methodology, improved performance, and could be explored in future work. We were also able to fine-tune for only 1-2 epochs, as we did not have access to higher computat… view at source ↗
Figure 2
Figure 2. Figure 2: Gaming community 7 CONCLUSION Our study aimed to test the hypothesis that fine-tuning a model on a community-specific dataset would enable the identification of slang, unique entities, and folklore from that community by examining words exhibiting the most significant semantic shifts in their embedding representations. Our experiments successfully supported this hypothesis. Furthermore, we observed that sl… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of models 3. Having more time to fine-tune our model and expand our vocabulary can help us perform masked language modeling better. 4. This method can be expanded further to de-bias current hate speech detectors and create systems that are more context-aware. ACKNOWLEDGMENTS We thank Professor Alan Ritter and Aman Khullar for their guidance and support. REFERENCES [1] James P. Bagrow… view at source ↗
Figure 6
Figure 6. Figure 6: t-SNE visualization for non-slang word embeddings [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
read the original abstract

We propose an unsupervised method of resolving slang, unique entities, and folklore from online communities by isolating words in the lexicon that have the highest magnitude of semantic shift. Semantic shift is defined as the evolution of a word's encoded representation as a result of fine-tuning a pretrained Large Language Model (LLM) on a community-specific text corpus. This value is inversely proportional to the cosine similarity between the base model's encoded representation of a word, and a fine-tuned model's encoded representation. We fine-tune the DistilRoBERTa model on text corpora collected from 3 Reddit subreddits (r/Technology, r/Gaming, r/WorldofWarcraft), model a distribution of cosine similarity over the lexicon, and show that one can successfully resolve words that have unique significance to the community by pulling data in the bottom 10-percentile. In contrast, we show that data in the top 10-percentile consist of words that carry relatively universal semantics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 7 minor

Summary. The paper proposes an unsupervised method for detecting community-specific slang, entities, and folklore by fine-tuning DistilRoBERTa on subreddit-specific corpora and measuring semantic shift as the cosine similarity drop between base and fine-tuned word embeddings. The authors apply the method to three subreddits (r/Technology, r/Gaming, r/WorldofWarcraft), report that the bottom 10-percentile of cosine similarity values yields community-specific terms, and present qualitative inspection of 15 words per community as primary evidence. Perplexity improvements (Table 2) confirm domain adaptation, while F1 scores (Table 1) are near zero and are dismissed by the authors.

Significance. The idea of using embedding shift between a base and fine-tuned model as an unsupervised signal for community-specific vocabulary is reasonable and has practical appeal for applications like hate-speech de-biasing. The paper is honest about its limitations and provides reproducible training hyperparameters (Table 2). However, the central claim is supported only by qualitative inspection of a small word list, with no quantitative precision/recall on the slang-detection task itself and no control for the most obvious confound (word frequency). The F1 scores in Table 1 are near zero and are dismissed rather than investigated, which weakens the empirical foundation.

major comments (4)
  1. §6, Table 1: The F1 scores for all models are near zero (0.019–0.064 for fine-tuned; 0–0.064 for baseline). The authors dismiss F1 as inappropriate for the task but do not replace it with any quantitative metric for the slang-detection claim. The central claim—that the bottom 10-percentile of cosine similarity 'successfully resolves' community-specific terms—is never evaluated with precision, recall, or any retrieval metric against the 90-word evaluation set. Without this, the claim is supported only by qualitative inspection of 15 words per community (§6, Figures 1–3), which is insufficient to establish that the method works as described.
  2. §5.2: The most critical untested confound is word frequency. Words that appear more frequently in the fine-tuning corpus receive more gradient updates and will naturally exhibit larger embedding shifts regardless of whether they are community-specific slang. The paper provides no correlation between corpus frequency and shift magnitude, no frequency-matched control, and no analysis showing that the bottom-percentile words are not simply high-frequency domain-general terms (e.g., 'game,' 'technology'). This is load-bearing for the central claim because the observed qualitative results are equally consistent with 'the method finds frequent domain words' as with 'the method finds slang.'
  3. §5.2: The embedding extraction method is underspecified. DistilRoBERTa produces contextual embeddings that vary by sentence context, yet the paper does not describe how per-word embeddings are aggregated from contextual representations (e.g., mean-pooling over occurrences, last-layer hidden state, specific layer, masking context). The cosine similarity computation—the core of the proposed method—cannot be reproduced or evaluated for correctness without this detail.
  4. §3.1.2: The evaluation set is constructed by manually selecting slang words from online community slang lists, which introduces a selection bias: the evaluation words are chosen precisely because they are known community-specific terms, so qualitative success at identifying them is partly expected. The paper does not report what fraction of the bottom-10-percentile retrieved words are in the evaluation set (precision), nor what fraction of the evaluation set is captured by the bottom-10-percentile (recall), making it impossible to assess retrieval quality.
minor comments (7)
  1. §3.2: The raw data statistics (3,848,330 posts, 22.08 GB) appear to describe the full Reddit dataset, but the per-community post counts (43,851 + 13,700 + 13,309 = ~71K) are much smaller. Clarify which statistics apply to the full dataset vs. the subset used.
  2. Table 1: The 'model' column labels are ambiguous—'WOW DistilRoBERTa' vs. 'DistilRoBERTa' does not clearly indicate fine-tuned vs. baseline. Add a header or footnote clarifying which row is which.
  3. Table 2: The baseline perplexity rows have inconsistent formatting (some have trailing dashes, one has an extra dash). Standardize the table presentation.
  4. §6: The text references 'Figure 4' as a table ('The table in Figure 4 showcases...'), but it is labeled as a figure. Clarify whether this is a table or figure and number accordingly.
  5. §2: 'Zhengqi Pe et al [6]' should be 'Zhengqi Pei et al.' (missing 'i' in the surname).
  6. §5.1: The claim that using a pretrained model makes the approach 'robust against' model shift from random initialization is stated but not tested. Either provide evidence or soften to a hypothesis.
  7. Figures 1–3: The axis labels and legend are not described in the caption. Clarify what the x/y axes represent and what the color coding means.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee correctly identifies that the central claim of the paper—that the bottom 10th percentile of cosine similarity values yields community-specific terms—lacks quantitative evaluation against the evaluation set, and that critical confounds (word frequency) and methodological details (embedding extraction) are not addressed. We agree with all four major comments and will revise the manuscript accordingly. Below we respond to each point.

read point-by-point responses

  1. Referee: §6, Table 1: F1 scores near zero, dismissed without replacement metric. Central claim never evaluated with precision/recall against the 90-word evaluation set.

    Authors: The referee is correct. We dismissed F1 without providing an alternative quantitative evaluation of the slang-detection claim itself. This is a genuine gap. In the revision, we will compute precision@k and recall@k for the bottom-10th-percentile retrieval against the 90-word evaluation set for each community. Specifically, we will report: (1) precision: the fraction of bottom-10th-percentile words that appear in the community-specific portion of the evaluation set, and (2) recall: the fraction of the evaluation set captured by the bottom-10th percentile. We will also report these metrics at multiple percentile thresholds (5th, 10th, 25th) to show how retrieval quality varies. This will replace the current purely qualitative evaluation as the primary evidence for the central claim. revision: yes

  2. Referee: §5.2: Word frequency is an untested confound. Frequent words receive more gradient updates and may exhibit larger shifts regardless of community-specificity. No frequency-matched control or correlation analysis provided.

    Authors: This is a fair and important concern. We will add a frequency analysis in the revision. Specifically, we will: (1) compute Spearman correlation between corpus frequency and cosine-similarity drop for each community, (2) inspect whether high-frequency domain-general terms (e.g., 'game,' 'technology') appear in the bottom-10th percentile, and (3) provide a frequency-stratified analysis showing the distribution of cosine-similarity drops across frequency bands. If frequency is a significant confound, we will report this transparently and discuss its implications. We note that the referee's concern is partially addressable by inspection of the current qualitative results: the bottom-10th-percentile words shown in Figures 1–3 include low-frequency proper nouns and community-specific terms (e.g., 'n'Zoth,' 'Azeroth') rather than high-frequency domain-general words, but this observation must be backed by the quantitative analysis we now commit to. revision: yes

  3. Referee: §5.2: Embedding extraction method underspecified. DistilRoBERTa produces contextual embeddings; paper does not describe aggregation method (mean-pooling, layer choice, masking, etc.).

    Authors: The referee is correct that this detail is missing and necessary for reproducibility. In the revision, we will specify the exact extraction procedure: for each target word, we extract the last-layer hidden state from DistilRoBERTa for every occurrence in the community corpus, mean-pool across all contextualized instances to obtain a single static vector per word, and compute cosine similarity between the base-model and fine-tuned-model vectors obtained by the same procedure. We will also report the number of occurrences per word used for aggregation and any minimum-frequency threshold applied. revision: yes

  4. Referee: §3.1.2: Evaluation set constructed by manually selecting known slang words, introducing selection bias. No precision or recall reported for the bottom-10th-percentile retrieval against the evaluation set.

    Authors: We agree on both points. The selection bias is real: by choosing known community-specific terms, we bias toward words the method is likely to find. We will address this in two ways in the revision. First, as noted in our response to the first comment, we will report precision and recall of the bottom-10th-percentile retrieval against the evaluation set, which at least quantifies how well the method recovers known slang. Second, we acknowledge in the revised text that this evaluation measures recall on known-positive terms but does not measure precision on the full retrieved set (since we lack negative labels for non-slang words in the bottom percentile). To partially address this, we will add a manual annotation of a random sample of bottom-10th-percentile words not in the evaluation set, labeling each as community-specific or not, and report annotated precision. We acknowledge that a fully unbiased evaluation would require an independent, externally validated slang lexicon, which does not currently exist for these communities. revision: partial

Circularity Check

0 steps flagged

No significant circularity: the method is unsupervised with no parameters fitted to evaluation data.

full rationale

The paper's core method—computing cosine similarity between base and fine-tuned DistilRoBERTa embeddings and pulling the bottom 10-percentile—is genuinely unsupervised. No parameter is fitted to the evaluation set; the 10-percentile threshold is an arbitrary cutoff, not a learned parameter. The evaluation set (§3.1.2) of 90 manually curated slang words is used only for the masked LM F1 task (which scores near zero, Table 1), not to calibrate the semantic shift retrieval. There are no self-citations; all references are to external work. The qualitative inspection of 15 words per community (§6, Figures 1–3) is weak evaluation but not circular: the method's output is not constructed from the evaluation labels. The genuine concerns—frequency confounds, lack of precision/recall on the retrieval task, unspecified embedding aggregation—are correctness and evaluation-quality issues, not circularity. The derivation chain does not reduce to its inputs by construction. Score 1 reflects one minor concern: the qualitative evaluation could be seen as confirmation-biased since the authors curated the evaluation set from online slang lists and then qualitatively inspected whether the method finds similar words, but this does not constitute formal circularity since no parameter is fit to make the 'prediction' match the input.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities, particles, or formal constructs. The free parameters are operational choices (percentile threshold, display count, epochs) rather than fitted constants. The axioms are domain assumptions about the relationship between semantic shift and community-specific language, none of which are independently verified.

free parameters (3)
  • Percentile threshold (10%) = 10
    The bottom 10-percentile of cosine similarity is chosen as the cutoff for slang detection (§1, §6). This threshold is selected post hoc by the authors, not derived from theory or optimized against a validation set.
  • Number of top/bottom words displayed = 15
    The authors display top 15 and bottom 15 words per community for qualitative evaluation (§6). This is an arbitrary choice.
  • Fine-tuning epochs = 1-2
    Limited to 1-2 epochs due to computational constraints (§7.1). This affects embedding quality but is not tuned.
axioms (4)
  • domain assumption Community-specific slang, entities, and folklore undergo greater semantic shift during fine-tuning than universal words.
    This is the core hypothesis stated in §1. It is assumed, not derived, and is the premise the entire method rests on.
  • domain assumption Cosine similarity between base and fine-tuned embeddings is a valid proxy for semantic shift magnitude.
    Stated in the Abstract and §5.2. While cosine similarity is a standard metric, its validity for measuring semantic shift in fine-tuned contextual models (as opposed to separately trained static embeddings) is not justified.
  • domain assumption DistilRoBERTa's masked language modeling captures community-specific semantics relevant to slang.
    §5.1 justifies the choice of DistilRoBERTa by its MLM training objective, but does not establish that MLM embeddings are optimal for slang detection vs. other approaches.
  • domain assumption Pre-trained model embeddings are stable enough that observed shifts are due to community-specific meaning rather than training noise.
    The paper claims robustness against randomization (§2, comparing to LSTM), but does not test this with multiple fine-tuning runs or seed variation.

pith-pipeline@v1.1.0-glm · 9866 in / 2574 out tokens · 156874 ms · 2026-07-05T16:47:39.799426+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references · 11 canonical work pages · 3 internal anchors

  1. [1]

    Information flow reveals prediction limits in online social activity

    James P. Bagrow, Xipei Liu, and Lewis Mitchell. “Information flow reveals prediction limits in online social activity”. In: Nature Human Behaviour 3.2 (Jan. 2019), pp. 122–128. issn: 2397-3374. doi: 10 . 1038 / s41562 - 018 - 0510 - 5.url: http : //dx.doi.org/10.1038/s41562-018-0510-5

    work page doi:10.1038/s41562-018-0510-5 2019

  2. [2]

    Semantic Variation in Online Communities of Practice

    Marco Del Tredici and Raquel Fernández. “Semantic varia- tion in online communities of practice”. In: arXiv preprint arXiv:1806.05847 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  3. [3]

    Mitigating Racial Biases in Toxic Lan- guage Detection with an Equity-Based Ensemble Frame- work

    Matan Halevy et al. “Mitigating Racial Biases in Toxic Lan- guage Detection with an Equity-Based Ensemble Frame- work”. In: Equity and Access in Algorithms, Mechanisms, and Optimization (Oct. 2021). doi: 10.1145/3465416.3483299. url: http://dx.doi.org/10.1145/3465416.3483299

    work page doi:10.1145/3465416.3483299 2021

  4. [4]

    Statistically significant detection of semantic shifts using contextual word embeddings

    Yang Liu, Alan Medlar, and Dorota Glowacka. “Statistically significant detection of semantic shifts using contextual word embeddings”. In:arXiv preprint arXiv:2104.03776 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  5. [5]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach

    Yinhan Liu et al. “Roberta: A robustly optimized bert pretrain- ing approach”. In: arXiv preprint arXiv:1907.11692 (2019). Julia Kruk, Sanchita Porwal, Amitrajit Bhattacharjee, and Mansi Phute Figure 6: t-SNE visualization for non-slang word embeddings in baseline and fine-tuned model

    work page internal anchor Pith review Pith/arXiv arXiv 1907

  6. [6]

    Slang detection and identification

    Zhengqi Pei, Zhewei Sun, and Yang Xu. “Slang detection and identification”. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) . 2019, pp. 881–889

    work page 2019

  7. [7]

    “DeepInf”

    Jiezhong Qiu et al. “DeepInf”. In:Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (July 2018). doi: 10.1145/3219819.3220077. url: http://dx.doi.org/10.1145/3219819.3220077

    work page doi:10.1145/3219819.3220077 2018

  8. [8]

    The Risk of Racial Bias in Hate Speech Detection

    Maarten Sap et al. “The Risk of Racial Bias in Hate Speech Detection”. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: As- sociation for Computational Linguistics, July 2019, pp. 1668–

    work page 2019

  9. [9]

    url: https://aclanthology

    doi: 10.18653/v1/P19-1163. url: https://aclanthology. org/P19-1163

    work page doi:10.18653/v1/p19-1163

  10. [10]

    Visualizing data using t-SNE

    Laurens Van der Maaten and Geoffrey Hinton. “Visualizing data using t-SNE.” In: Journal of machine learning research 9.11 (2008)

    work page 2008

  11. [11]

    Digital begriffsgeschichte: Tracing semantic change using word embeddings

    Melvin Wevers and Marijn Koolen. “Digital begriffsgeschichte: Tracing semantic change using word embeddings”. In:Histor- ical Methods: A Journal of Quantitative and Interdisciplinary History 53.4 (2020), pp. 226–243

    work page 2020