REVIEW 4 major objections 5 minor 30 references
Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Large language models can define queer slang reliably when given an example sentence or a domain cue, but without either they fall back on the word's everyday meaning.
desk verdict A genuinely useful new dataset and a plausible first result on LLMs and queer slang, but the quantitative core is thin and the single-annotator labeling is the load-bearing risk. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Slang-Q: a manually curated dataset of 1,024 user-generated English sentences, each paired with one of 118 queer-related terms and a reference definition, organized by a two-level taxonomy with broad categories (Identity, Slang, Intersectional) and optional subcategories (Reclaimed, Shorthand, Idiomatic expression, Pronoun, Spelling variation). The evaluation machinery is the crossed design of two binary prompt dimensions—domain framing (generic language expert versus queer internet slang expert) and context (term only versus term plus example sentence)—which isolates the effect of each on definition generation. Scores are computed with a lexical-overlap metric and a semantic-similarity metric against three reference definitions per term, then aggregated at the term level so that frequent terms do not dominate the results.
What would settle it
Independently re-annotate a random sample of the 1,024 sentences with several annotators who are familiar with online queer slang, and measure inter-annotator agreement; if a substantial share of the queer-sense and harmfulness labels are not reproduced, the gold standard and the prompt-condition scores built on it collapse.
Extended reading notes
Core claim
On its own terms, the central discovery is that what an LLM knows about queer slang is not a fixed quantity: it is strongly conditioned by whether the prompt identifies the domain and whether an example sentence is present. Across all four evaluated models, the bare-term baseline scores lowest on both automatic metrics, while slang-informed framing alone and contextual grounding alone each improve scores, and the two together perform best. Manual inspection of 200 outputs from the two conditions with the strongest contrast supports this: in the bare-term condition, roughly a quarter to a third of definitions miss the intended sense, while in the slang-informed context condition the share of fully incorrect definitions drops to zero in one model and to a small fraction in the other, though some definitions still omit sociocultural nuance. The paper also finds that all models fall below the human reference bound, that no model dominates, and that intersectional terms—those shared with other communities such as AAVE or fandom—are the hardest category. A data-contamination probe finds that one proprietary model appears to have memorized far more of the source sentences than the open-weight models, yet this does not translate into better definitional performance.
Load-bearing premise
The whole evaluation rests on one author's manual judgment, without measured inter-annotator agreement, of which sentences use each term in its intended queer sense and which content is harmful; if those labels are wrong or inconsistent, the gold definitions and all model scores built on them shift.
Editorial extensions
If this is right
- A model asked to define a slang term in isolation will make noticeably more errors than the same model given a single example sentence, so user-facing systems that explain community slang should avoid bare-term queries.
- Explicitly telling the model that it is dealing with queer slang is enough by itself to shift some definitions toward the queer-specific sense, and combining that framing with an example sentence yields the strongest correctness in the manual evaluation.
- Intersectional terms that circulate in multiple communities, such as AAVE or fandom, are the hardest to define, meaning queer-slang evaluation should treat multi-community usage as a first-class difficulty rather than noise.
- Since all four tested models cluster within a narrow performance band, the results point to broadly similar queer-slang knowledge across model families and sizes, with none reaching the human reference bound.
- The Slang-Q dataset provides a reusable evaluation set for future work on definition generation, prompt design, and slang handling in large language models.
Reading between the lines
- Because the model with the highest estimated memorization of the source corpus does not outperform the others, the paper's data point implicitly suggests that exposure to slang-heavy text during training is not the limiting factor for community-specific understanding; prompt and context design may be more actionable.
- The taxonomy's 'Intersectional' label acknowledges that queer slang overlaps with AAVE and fandom usage; a fairer scoring protocol might compare model definitions against multiple gold definitions, one per community, since a single reference definition will penalize valid senses.
- A direct testable extension is to apply the same two-by-two prompt design to other dynamic sociolects, such as disability community language or regional slang, to see whether the context-and-framing effect generalizes beyond queer slang.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Slang-Q, a dataset of 1,024 naturally occurring English sentences paired with 118 queer slang terms and reference definitions drawn from external lexicons, together with a taxonomy of queer slang. Using this resource, the authors evaluate four LLMs (Claude Sonnet 4.6, LLaMA 3.3 70B, LLaMA 4 Scout, Qwen3 32B) on a definition-generation task under four prompt conditions that vary whether the prompt is slang-informed and whether an example sentence is provided. Automatic evaluation with ROUGE-L and BERTScore is supplemented by a manual error analysis on two models. The paper concludes that models fall below a human upper bound, that withholding domain framing and sentential context hurts performance, and that both slang-informed prompting and contextual grounding help models converge on the intended queer-specific meaning.
Significance. If the results are reliable, this is a useful contribution to an underserved area: queer slang is underrepresented in NLP, and Slang-Q could become a reference resource for evaluating LLM understanding of community-specific language. The authors are transparent about data sources, make the repository available, include a data-contamination check, and provide a manual error analysis that partially corroborates the automatic findings. However, the current evidence is exploratory: the automatic metric differences are small, the 'human' bound is not a true human-performance measurement, and the entire dataset rests on single-annotator judgments without reliability checks. These issues are central to the paper's headline claims rather than cosmetic.
major comments (4)
- [Section 3.3] The annotation was carried out by the first author alone, with only 'fewer ambiguous cases' discussed with the second author, and no inter-annotator agreement or reliability statistic is reported. Because every sentence-term pairing, every judgment that a term is used in its intended queer sense, and every harmfulness decision is single-annotator, the two headline findings--that models score below a human bound and that context/framing improve performance--inherit any systematic bias in these labels. If the annotator retained sentences in which the queer sense is strongly disambiguated by context and discarded genuinely ambiguous ones, the context-benefit effect could be inflated or even manufactured. The manuscript should report a reliability study (e.g., a second annotator on a stratified sample with Cohen's kappa or equivalent) and an adjudication protocol, or the conclusions must be substantially softened.
- [Section 5, Table 5] The row labeled 'Human' is not a measure of human performance; it is the mean similarity of the gold reference definition to the two GPT-5.5-generated alternative definitions. The conclusion in Section 6 that 'models fall below the human upper bound' is therefore not supported by the data as presented. Either collect and score actual human-written definitions for a sample of terms, or relabel this quantity as a reference-agreement ceiling and revise the wording of the finding accordingly.
- [Section 5, Table 5] The reported condition differences are very small relative to the standard deviations (e.g., ROUGE-L 0.16 vs 0.18, BERTScore 0.85 vs 0.86, with SDs around 0.05-0.07), and no significance tests are reported. The claim in Section 6 that withholding domain framing and sentential context 'consistently hurts performance' requires paired significance testing across terms (e.g., Wilcoxon signed-rank or bootstrap) and effect sizes. The manual evaluation is suggestive but covers only two models and two conditions, so it cannot by itself establish the cross-model claim.
- [Section 4.4 and Section 5] Claude Sonnet 4.6 shows a high verbatim contamination range (57.30-62.00%) with the source corpus, and the context condition supplies the exact example sentences from that corpus. Although the authors argue that memorization of a sentence does not entail ability to solve the definition task, memorized examples could plausibly inflate performance specifically in the context conditions. Please provide a per-model breakdown of the context benefit and discuss how contamination affects the interpretation, or explicitly control for it in the analysis.
minor comments (5)
- [Section 3.3] The discard counts (1,833 semantically unrelated + 327 harmful/vulgar = 2,160) leave 1,024 of 3,184 sentences; stating the retained percentage explicitly (about 32%) would prevent reader confusion.
- [Section 5] There is a typo in 'showes' (should be 'shows'), and the condition naming is inconsistent between 'Term baseline' in the text and 'Term (Base)' in the first result paragraph.
- [Table 5] The 'Human' row would benefit from a footnote stating explicitly that it measures agreement among the gold and paraphrased reference definitions, not human-written outputs.
- [Figures 1 and 2] The category panels aggregate across all models and conditions, which can mask important interactions; adding error bars or significance markers would make the visual claims more interpretable.
- [Section 6] The limitations paragraph should mention the single-annotator reliability issue and the lack of significance tests, since these are the main threats to the stated conclusions.
Circularity Check
No circularity: model scores are compared against externally sourced gold definitions and the dataset is not derived from the evaluated models.
full rationale
The paper's central claim compares model-generated definitions against gold references assembled from external lexical sources (Lexicon Library.LGBT, Wiktionary, Wikipedia, and prior NLP inventories), not from the evaluated models: "When a term is added from a lexical source, such as Lexicon Library.LGBT, Wiktionary, or Wikipedia's LGBTQ slang category, we use the definition provided by that source" (§3.1.1). The GPT-5.5-generated alternative references were manually validated for meaning preservation and are used uniformly as additional references, so they do not force the observed condition effects. The only self-citations (e.g., Savoldi et al. [3] and Piergentili et al. [18]) are background references, not load-bearing evidence for the dataset or the evaluation conclusions. The single-annotator labeling described in §3.3 ("Annotation was carried out by the first author") is a genuine validity and reliability risk, but it is not circular: the annotation decisions determine which sentences enter the dataset, not the models' scores by construction, and the paper explicitly notes that contamination does not entail task-solving ability (§4.4). Since no fitted parameter is renamed as a prediction and no claim reduces to its own input by definition, the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (5)
- domain assumption Urban Dictionary sentences extracted from the SLANG benchmark are a suitable and representative source of naturally occurring queer slang in user-generated content.
- domain assumption ROUGE-L and BERTScore, averaged over the gold definition and two GPT-5.5 paraphrases, are valid proxies for definition quality.
- domain assumption The single annotator's classification of intended queer sense, harmfulness, and taxonomy category is reliable enough to serve as ground truth.
- domain assumption The 'human upper bound' measured as gold-vs-GPT-5.5-paraphrase similarity is a meaningful comparator for model performance.
- domain assumption Verbatim contamination detected by DCQ does not benefit the definition-generation task.
Cite this review
Pith. "Pith review of Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content." pith.science (2026). https://pith.science/paper/XPSFPSTL
@misc{pith2026260804847,
author = {Pith},
title = {Pith review of: Do Language Models Know Their Slang? Queer Slang Understanding in User-Generated Content},
year = {2026},
howpublished = {\url{https://pith.science/paper/XPSFPSTL}},
note = {Machine review of arXiv:2608.04847}
}
read the original abstract
Despite its cultural relevance and diffusion, queer slang remains underrepresented in Natural Language Processing research. Towards addressing this gap, we introduce Slang-Q, a manually curated dataset of naturally user-generated English sentences paired with queer slang terms and reference definitions, built upon a newly constructed taxonomy of 118 queer terms. We use this resource to conduct a first exploratory evaluation of language models on their ability to understand and define queer slang under varying prompting conditions. Slang-Q is intended as a basis for studying how current models handle sensitive, community-specific language and whether they can provide accurate and reliable information about such forms of identity and linguistic expression.
Figures
Reference graph
Works this paper leans on
-
[1]
C. Chen, K. Shu, Combating misinformation in the age of llms: Opportunities and challenges, AI magazine 45 (2024) 354–368
work page 2024
-
[2]
A. Chatterji, T. Cunningham, D. J. Deming, Z. Hitzig, C. Ong, C. Y. Shan, K. Wadman, How people use chatgpt, Technical Report, National Bureau of Economic Research, 2025
work page 2025
-
[3]
B. Savoldi, G. Attanasio, O. Gorodetskaya, M. M. Manerba, E. Bassignana, S. Casola, M. Negri, T. Caselli, L. Bentivogli, A. Ramponi, A. Muti, N. Balbo, D. Nozza, Generative ai practices, literacy, and divides: An empirical analysis in the italian context, 2026. URL: https://arxiv.org/abs/2512. 03671.arXiv:2512.03671
arXiv 2026
-
[4]
J. Basoah, D. Chechelnitsky, T. Long, K. Reinecke, C. Zerva, K. Zhou, M. Díaz, M. Sap, Not like us, hunty: Measuring perceptions and behavioral effects of minoritized anthropomorphic cues in llms, in: Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, 2025, pp. 710–745
work page 2025
-
[5]
L. Mei, S. Liu, Y. Wang, B. Bi, X. Cheng, SLANG: New concept comprehension of large language models, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 12558–12575. URL: https://aclanthology.org/20...
-
[6]
C. C. Eble, Slang & sociability: In-group language among college students, Univ of North Carolina Press, 1996
work page 1996
-
[7]
Kulick, Gay and lesbian language, Annual review of anthropology 29 (2000) 243–285
D. Kulick, Gay and lesbian language, Annual review of anthropology 29 (2000) 243–285
work page 2000
-
[8]
H. Motschenbacher, Taking queer linguistics further: Sociolinguistics and critical heteronormativ- ity research., International journal of the sociology of language 2011 (2011)
work page 2011
Show all 30 references
-
[9]
Cifor, K
M. Cifor, K. Rawson, Mediating queer and trans pasts: The homosaurus as queer information activism, Information, Communication & Society 26 (2023) 2168–2185
2023
-
[11]
R. Dorn, L. Kezar, F. Morstatter, K. Lerman, Harmful speech detection by language models exhibits gender-queer dialect bias, in: Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 2024, pp. 1–12
2024
-
[12]
Vásquez, G
J. Vásquez, G. Bel-Enguix, S. T. Andersen, S.-L. Ojeda-Trueba, Heterocorpus: A corpus for heteronormative language detection, in: Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), 2022, pp. 225–234
2022
-
[13]
Tint, Guardrails, not guidance: Understanding responses to LGBTQ+ language in large language models, in: A
J. Tint, Guardrails, not guidance: Understanding responses to LGBTQ+ language in large language models, in: A. Pranav, A. Valentine, S. Bhatt, Y. Long, A. Subramonian, A. Bertsch, A. Lauscher, A. Gupta (Eds.), Proceedings of the Queer in AI Workshop, Association for Computatio...
2025 doi
-
[14]
Weber, A
S. Weber, A. Wang, A. Gupta, A. Subramonian, D. Ulmer, E. Tanwar, G. Aich, H. Devinney, J. Hobbs, J. Mickel, J. Tint, M. Sosto, R. Groshan, S. Astarita, V. Gautam, V. Blaschke, W. Agnew, W. Y. Lee, Y. Long, Queer nlp: A critical survey on literature gaps, biases and trends, 20...
2026 arXiv
-
[15]
Sosto, D
M. Sosto, D. S. M. Pandiani, L. Hollink, QueerGen: How LLMs reflect societal norms on gender and sexuality in sentence completion task, in: V. Demberg, K. Inui, L. Marquez (Eds.), Findings of the Association for Computational Linguistics: EACL 2026, Association for Computation...
2026 doi
-
[16]
URL: https://lexicon.library.lgbt/
Library.LGBT, Lexicon library.lgbt, n.d. URL: https://lexicon.library.lgbt/
-
[17]
Veloso, L
L. Veloso, L. Hirlimann, P. Wicke, H. Schütze, Slaying: Towards queer language processing, 2025. URL: https://arxiv.org/abs/2509.17449.arXiv:2509.17449
2025
-
[18]
Piergentili, B
A. Piergentili, B. Savoldi, M. Negri, L. Bentivogli, Enhancing gender-inclusive machine translation with neomorphemes and large language models, in: C. Scarton, C. Prescott, C. Bayliss, C. Oakley, J. Wright, S. Wrigley, X. Song, E. Gow-Smith, R. Bawden, V. M. Sánchez-Cartagena...
2024
-
[19]
E. L. Ungless, S. Dev, C. L. Bennett, R. Gulotta, J. Bastings, R. Denton, Amplifying trans and nonbinary voices: A community-centred harm taxonomy for LLMs, in: W. Che, J. Nabende, E. Shutova, M. T. Pilehvar (Eds.), Proceedings of the 63rd Annual Meeting of the Association for...
2025
-
[20]
Locatelli, G
D. Locatelli, G. Damo, D. Nozza, A cross-lingual study of homotransphobia on Twitter, in: S. Dev, V. Prabhakaran, D. I. Adelani, D. Hovy, L. Benotti (Eds.), Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), Association for Computational Linguis...
2023
-
[21]
Pofcher, C
J. Pofcher, C. M. Homan, R. Sell, A. R. KhudaBukhsh, Hope vs. hate: Understanding user interactions with LGBTQ+ news content in mainstream US news media through the lens of hope speech, in: C. Christodoulopoulos, T. Chakraborty, C. Rose, V. Peng (Eds.), Proceedings of the 2025...
2025 doi
-
[22]
Ghosal, A
A. Ghosal, A. Gupta, V. Srikumar, Unequal voices: How llms construct constrained queer narratives,
-
[23]
Ramesh, S
K. Ramesh, S. Kumar, A. Khudabukhsh, Revisiting queer minorities in lexicons, in: K. Narang, A. Mostafazadeh Davani, L. Mathias, B. Vidgen, Z. Talat (Eds.), Proceedings of the Sixth Work- shop on Online Abuse and Harms (WOAH), Association for Computational Linguistics, Seat- t...
2022 doi
-
[24]
Subramonian, V
A. Subramonian, V. Gautam, P. Seshadri, D. Klakow, K.-W. Chang, Y. Sun, Agree to disagree? a meta- evaluation of llm misgendering, 2025. URL: https://arxiv.org/abs/2504.17075.arXiv:2504.17075
2025 arXiv
-
[25]
Team, Qwen3 technical report, 2025
Q. Team, Qwen3 technical report, 2025. URL: https://arxiv.org/abs/2505.09388. arXiv:2505.09388
2025 arXiv
-
[26]
by arXiv, The llama 4 herd: Architecture, training, evaluation, and deployment notes, 2026
R. by arXiv, The llama 4 herd: Architecture, training, evaluation, and deployment notes, 2026. URL: https://arxiv.org/abs/2601.11659.arXiv:2601.11659
2026
-
[27]
Grattafiori, A
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sra- vankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spatar...
2024 arXiv
-
[28]
Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp
C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013/
2004
-
[29]
Zhang, V
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, 2020. URL: https://arxiv.org/abs/1904.09675.arXiv:1904.09675
2020 arXiv
-
[30]
A Chinese internet slang term referring to a person whose sex- ual orientation or gender iden- tity is perceived as fluid or am- biguous
S. Golchin, M. Surdeanu, Data contamination quiz: A tool to detect and estimate contamination in large language models, Transactions of the Association for Computational Linguistics 13 (2025) 809–830. URL: https://aclanthology.org/2025.tacl-1.37/. doi:10.1162/tacl.a.20. A. Ter...
2025 doi
-
[2025]
URL: https://arxiv.org/abs/2507.15585.arXiv:2507.15585
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.