{"id":"71dec895-4546-44f0-b904-2296f901503d","arxiv_id":"2501.16507","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Using a new taxonomy and a RAG-enhanced LLM classifier on TikTok, this paper finds anti-trans accounts outnumber and heavily interact with pro-trans accounts, but the classifier's evaluation is not shown to be leak-free.","lead":"This paper introduces a taxonomy of anti- and pro-trans TikTok content, trains an LLM-based classifier with retrieval augmentation, and maps reply networks between the two communities. It reports that anti-trans accounts outnumber pro-trans accounts in these interactions and engage across community lines, suggesting targeted harassment, though the evaluation design may overstate accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Classification evaluation is not held out: the same 300 annotated samples are used to select prompts, populate the RAG database, and compute accuracy, so the reported metrics and the network labels derived from them are unvalidated.","rationale":"We agree with the reader's identification of the weakest link. The paper's three contributions (taxonomy, RAG classifier, network analysis) are coupled: the network analysis uses the classifier's output as node labels, so the classifier's validity gates the central claim. The reported classification results cannot be trusted because all annotator labels are used for prompt selection and as RAG retrieval contexts, and the same labels are then scored; this is textbook test-set leakage. No alternative evaluation (e.g., a temporally or randomly held-out set) is reported, and the code/datasheet are only linked, not released in the paper. The leakage concern is concrete and independently checkable: fixing it would require re-running the pipeline with a proper split. We also note that even with clean labels, the 'targeting' inference would require directionality or content analysis of the edges; low assortativity alone does not distinguish anti-trans actors attacking pro-trans users from pro-trans users engaging in refutation (a category in the paper's own taxonomy). That said, the classification-leakage issue is the more fundamental and falsifiable concern, and it alone is sufficient to reject the empirical network findings as currently presented. The taxonomy and ethical framing remain useful contributions, but the central empirical claim is not supported.","tokens_in":17352,"tokens_out":5930,"duration_ms":55912,"concrete_test":"Perform a strict 5-fold cross-validation on the 300 annotated samples: for each fold, exclude the validation fold from prompt selection and from the RAG retrieval database (and deduplicate near-duplicate examples), then report macro-F1 and per-class precision/recall on the held-out fold. Then re-label the largest connected components in Section 5.2 with a model trained only on the training folds and recompute the anti-to-pro ratios and assortativity. If held-out anti-trans precision remains near 0.37 or the ratios change substantially, the network conclusions in Section 5.2 do not survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central network claim (Section 5.2, abstract) depends on labels assigned by the LLaMA3+RAG+Taxonomy classifier to nodes in the reply graphs. That classifier is never evaluated on held-out data. Section 4.4.1 states that of 8 prompts, 'we also kept track of which prompt performed the best over all samples, and it was that prompt we used' — i.e., prompt selection uses the entire 300-sample annotated set. Section 4.4.2 indexes 'annotated examples' into the RAG retrieval database, and Section 5.1 reports accuracy/F1 on the same n=300 samples (Anti n=52, Pro n=160, Neutral n=88). With the retrieval database containing annotated examples, a test item's own label (or a near duplicate) can be retrieved as context, inflating accuracy. No train/test split, cross-validation, or temporal separation is described. Because Section 5.2 then applies this same pipeline to label unannotated network samples, the reported anti-to-pro ratios (5:1 tags/replies, 2.5:1 duets/stitches) and the very low assortativity values rest on unvalidated labels. This is compounded by the model's low precision for the anti-trans class (0.37 in Table 2): false positives on neutral content are explicitly acknowledged in Section 6.1, so the anti-trans node population, which drives the headline ratio, is likely inflated. The targeting claim therefore lacks a trustworthy labeling foundation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a taxonomy of pro-trans, anti-trans, and neutral sentiment on TikTok, builds an LLM-based classifier that combines Retrieval-Augmented Generation (RAG) with annotated examples and taxonomy definitions, and applies this classifier to label reply networks among TikTok users. The authors report that anti-trans users outnumber pro-trans users by about 5:1 in tags/replies and 2.5:1 in duets/stitches, and that assortativity is very low (reply = -0.79), interpreting this as evidence that anti-trans actors target pro-trans users. The paper also discusses the taxonomy, annotation process, classification results, and limitations.","tokens_in":17680,"tokens_out":4354,"duration_ms":37165,"significance":"If the classification pipeline and network analysis were methodologically sound, the paper would make a useful contribution: the taxonomy is grounded in relevant sociological and gender-studies literature, the RAG approach is interesting, and the network-level characterization of anti-trans and pro-trans interactions on TikTok addresses an important and understudied topic. The authors also provide ethical safeguards, a datasheet, and a positionality statement. However, the validity of the empirical claims, especially the central network finding, depends entirely on the classifier label quality, and the current evaluation does not establish that validity.","major_comments":[{"comment":"The reported classification accuracy is not a valid estimate of generalization because the prompt selection uses all 300 labeled samples. Section 4.4.1 states that 'we also kept track of which prompt performed the best over all samples, and it was that prompt we used for the following approaches.' Section 5.1 then evaluates the model on the same 300 samples. This means the best prompt is selected using test labels, so the improvement shown in Table 2 (e.g., accuracy 0.47 to 0.67) is confounded by prompt selection on the evaluation set.","section":"§4.4.1 and §5.1"},{"comment":"No train/test split is described for the RAG retrieval database. In Section 4.4.2, annotated examples are indexed into the retrieval database, and Section 5.1 reports performance on the same n=300 annotated set. If a test item itself, or a near-duplicate, is retrieved as context, the model may effectively memorize the label. The paper does not state that test items are excluded from the retrieval database, so the reported accuracy of the RAG models is potentially circular and cannot be used to support the subsequent network labels.","section":"§4.4.2 and §5.1"},{"comment":"The network analysis depends on labels produced by the LLaMA3+RAG Examples+RAG Taxonomy classifier applied to unannotated network samples, but that classifier has precision of only 0.37 for the anti-trans class (Table 2) and the authors acknowledge in Section 6.1 that the model has a larger false positive rate for anti-trans classification, flagging neutral content as anti-trans. Consequently, the anti-trans node population in Figure 4 is likely inflated, which directly undermines the reported anti-to-pro ratios (5:1 and 2.5:1) and the low assortativity values. The central network claim in the abstract and Section 6.2 therefore lacks a trustworthy labeling foundation.","section":"§5.2 and §6.1"},{"comment":"The sublabel recall values are based on very small counts. For example, the INTRA sublabel has a proportion of 0.01 of the sample, which corresponds to roughly 3 instances in the 300-sample set, yet the model reports recall of 1.00 for this sublabel in the RAG Samples model. Such values are not meaningful and should be reported with confidence intervals or excluded. This issue applies to several other sublabels with proportion ≤ 0.07 (e.g., XOR, TERF).","section":"Table 2"},{"comment":"The abstract describes 'hired expert data annotators from the trans/nonbinary community,' but Section 4.3.2 states that the annotation team consists of two internal researchers who are members of the trans/non-binary community. This is a factual inconsistency that affects the paper's claims about external annotation expertise and should be corrected.","section":"Abstract and §4.3.2"}],"minor_comments":[{"comment":"The sentence 'and that Results from network analysis indicate...' contains an awkward capitalization/grammar error; 'Results' should not be capitalized mid-sentence.","section":"Abstract"},{"comment":"The description of the network experiments is incomplete: the paper does not report the number of nodes and edges in the largest connected components, the proportion of nodes with labels, or how neutral nodes are treated when computing assortativity. These details are needed to interpret the network statistics.","section":"§5.2"},{"comment":"The name 'LLaMA3+RAG Examples+RAG Taxonomy' is inconsistent with the model name 'LLaMA3+RAG Samples+RAG Taxonomy' used in Table 2; please standardize the notation.","section":"§4.4.3"},{"comment":"Several references have duplicated author names, for example reference [21] lists 'Claudiu Gabriel Ionescu and Monica Licu' correctly but reference [29] repeats 'Ellen Simpson, Ellen Simpson' and 'Bryan Semaan, Bryan Semaan.' These should be cleaned up.","section":"References"},{"comment":"The paper should specify how disagreements between the two annotators were resolved, given the reported Cohen's Kappa of 0.64 on the 50-sample subset.","section":"§4.3.2"}],"recommendation":"major_revision","confidential_remarks":"The reviewer's concern about the lack of a held-out evaluation is fully borne out by the manuscript. The classification evaluation, RAG retrieval setup, and network labeling are all entangled on the same 300 annotated samples, so the empirical results cannot be considered valid as reported. The taxonomy and the overall research question are valuable, but the paper needs a substantial methodological rework—proper train/test splitting, exclusion of test items from the retrieval database, and a validation of the network labels—before it can be considered for publication. I would not recommend rejection outright because the flaws are fixable in principle, but the revision required is substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for its taxonomy, not for its network numbers. The authors build a careful, theory-grounded taxonomy of pro-trans, anti-trans, and neutral TikTok content, with subcategories like exorsexism and anti-transmasculinity that earlier work misses, and they involve trans/nonbinary community experts in labeling 300 samples. That part is genuinely useful and could support future annotation efforts. The positionality and ethics discussion is also thoughtful. But the classification evaluation has a load-bearing leakage problem, and since every network claim flows through that classifier, the headline result—anti-trans users outnumbering pro-trans users about 5:1 with very low assortativity as evidence of targeting—is not currently supported.\n\nThe paper never describes a train/test split. Section 4.4.1 selects the best prompt using all 300 labeled samples. Section 4.4.2 indexes annotated examples into the RAG retrieval database. Section 5.1 then reports accuracy on the same 300 samples. If a test item can retrieve its own labeled example as context, the reported gains are partly memorization, not generalization. The anti-trans false-positive rate is also high (precision 0.37), which likely inflates the anti-trans node counts in the reply graphs. Sublabel recall numbers rest on tiny counts—one category has only 1% of samples—so those should not be read as stable. The paper does not release code or full artifacts either, which makes the pipeline hard to check.\n\nNone of this means the taxonomy is wrong, and the ethnographic and sociological framing is generally solid. The citation pattern looks fine, including the self-citations to related work on gender-queer dialect bias. But the central empirical claim, stated in the abstract as demonstrating targeting of trans individuals, depends entirely on classifier labels that have not been validly evaluated.\n\nThis deserves a serious referee—the taxonomy and the question are worth engaging—but it should go back for major revision: prompt selection on a validation split, retrieval from a training split only (or leave-one-out), and evaluation on unseen data. If the authors can show the classifier generalizes, the network analysis could become credible. As it stands, the paper is a useful conceptual contribution attached to an unvalidated measurement.","headline":"Valuable taxonomy, but the classifier is evaluated on the same 300 samples used to build and tune it, so the network findings that rest on those labels are not yet supported.","tokens_in":18163,"tokens_out":2283,"would_cite":true,"duration_ms":22586,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Anti-trans TikTok users disproportionately direct replies, tags, duets, and stitches at pro-trans users.","keywords":["TikTok","Online Harassment","Computational Social Science","Classification","Network analysis","Retrieval-Augmented Generation (RAG)","Large language models (LLMs)","Community Dynamics"],"falsifier":"Run a strict train/test evaluation: split the 300 annotated videos into disjoint training and test sets, build the RAG retrieval database only from the training set, select the best prompt on the training set, report accuracy on the test set, then recompute the reply-network assortativity and the anti-to-pro interaction ratios using the corrected node labels. If test-set accuracy drops materially below 0.67 or the network statistics lose their strong negative assortativity, the targeting conclusion does not survive.","tokens_in":17194,"feed_emoji":"🏳️‍⚧️","tokens_out":9620,"duration_ms":73862,"temperature":0.7,"pith_summary":"The paper tries to establish that anti-trans actors on TikTok are not sealed off in their own echo chamber: they disproportionately direct replies, tags, duets, and stitches at pro-trans users, a pattern the authors interpret as targeting of trans individuals. To reach this conclusion, the authors build a three-tier taxonomy of sentiment toward trans and nonbinary people, hire trans/nonbinary annotators to label 300 TikTok videos, and feed the taxonomy plus annotated examples into a retrieval-augmented generation (RAG) classifier built on LLaMA 3, which then labels the larger unannotated video set and its interaction networks. The network analysis shows anti-trans users outnumber pro-trans users about 5 to 1 in tag/reply interactions and 2.5 to 1 in duet/stitch interactions, with reply-network assortativity of −0.79 (−0.93 when neutral nodes are excluded). The same pipeline yields a secondary result: adding the taxonomy to the RAG prompts raises overall classification accuracy from 0.47 (zero-shot LLaMA 3) to 0.67, and nearly triples recall for pro-trans content. If the network finding holds, it points to concrete moderation targets: anti-trans accounts that initiate cross-community interactions.","feed_headline":"Anti-trans TikTok users target pro-trans creators 5-to-1 in replies","feed_subtitle":"Low assortativity and 5-to-1 reply ratios lead the authors to conclude anti-trans actors are targeting trans users.","key_machinery":"The argument is carried by three coupled components. The first is a sentiment taxonomy that classifies content as Pro-Trans, Anti-Trans, or Neutral, with anti-trans subcategories for transmisogyny, anti-transmasculinity/transandrophobia, exorsexism, TERF, right-wing, and intracommunity sentiment, and pro-trans subcategories such as celebration of trans existence and refutation of anti-trans rhetoric. The second is a retrieval-augmented generation (RAG) pipeline: LLaMA 3 classifies a TikTok's transcription and description, and the prompt is augmented by retrieving relevant annotated examples and taxonomy definitions from an index, which measurably improves accuracy. The third is network analysis of four TikTok interaction types—tags, replies, duets, and stitches—where node labels come from the classifier and edge structure is summarized by assortativity and by the ratio of anti-trans-to-pro-trans edges; the very negative assortativity values are what license the targeting interpretation.","core_discovery":"The central claim is that the reply networks connecting pro-trans and anti-trans TikTok creators are strongly disassortative: anti-trans actors are far more likely to initiate interaction with pro-trans users than with each other, while pro-trans users tend to appear as isolated nodes or in tight-knit clusters. The authors read this structural pattern as evidence of targeting—anti-trans actors attacking trans individuals—rather than two communities simply ignoring each other. The paper's supporting discovery is that its taxonomy-enhanced RAG classifier distinguishes pro-trans, anti-trans, and neutral content better than the base LLaMA 3 model, particularly by recovering 'celebration of trans existence' content that the base model mislabels as neutral; overall accuracy rises from 0.47 to 0.67 when annotated examples and taxonomy definitions are retrieved and appended to the prompt.","pith_inferences":["The hashtag-seeded snowball sample skews toward activist-adjacent and dog-whistle content; a replication sampling TikTok's For You feed without hashtag filters would test whether the 5-to-1 ratio and negative assortativity generalize beyond hashtag-driven communities.","Because the paper labels videos but not the direction of the initiating action at edge level, a finer-grained audit that verifies which user created the duet, stitch, or reply would strengthen the causal reading that anti-trans actors are the aggressors rather than the responders.","The taxonomy's sublabels could drive a follow-up analysis of which specific anti-trans categories (e.g., transmisogyny versus exorsexism) account for most cross-community edges, enabling targeted interventions rather than whole-community moderation.","The same pipeline could be applied to other short-form video platforms (Instagram Reels, YouTube Shorts) to test whether the targeting structure is a TikTok-specific algorithmic outcome or a general property of short-form video ecosystems."],"forward_implications":["If the network finding holds, content moderation systems should prioritize anti-trans accounts that repeatedly initiate replies, duets, and stitches with pro-trans creators, since those edges are the primary site of cross-community harassment.","The taxonomy gives platforms a finer-grained vocabulary—transmisogyny, anti-transmasculinity, exorsexism—so automated systems can distinguish distinct harms instead of lumping all anti-trans content together.","The RAG-with-taxonomy result suggests that expert-curated definitions and examples can substantially improve LLM classification of identity-related marginal content without fine-tuning, an approach that transfers to other marginalized communities.","The low assortativity values imply anti-trans content bridges into pro-trans and neutral spaces rather than circulating in a segregated network, which may explain why trans users report being served content that maligns their identity.","The biggest remaining error mode is over-flagging neutral content as anti-trans (anti-trans precision drops to 0.37 in the taxonomy model), so delineating neutral from anti-trans content is a concrete next target."],"supporting_citations":[{"why":"provides the RAG methodology the classification pipeline is built on.","marker":"[33]"},{"why":"supplies LLaMA 3, the base LLM whose zero-shot performance is the baseline.","marker":"[28]"},{"why":"supplies Whisper, the speech-to-text model that turns video audio into the classification input.","marker":"[68]"},{"why":"documents the TikTok Research API used for data collection.","marker":"[22]"},{"why":"documents the video-download failures (private videos, photo collages, Pyktok failures) that shape the dataset.","marker":"[65]"},{"why":"guides the annotation ethics and protocol used to label the 300 training samples.","marker":"[24]"},{"why":"provides the hope-speech and counter-speech concepts that ground the pro-trans taxonomy categories.","marker":"[18]"},{"why":"defines transmisogyny, a load-bearing anti-trans subcategory in the taxonomy.","marker":"[74]"},{"why":"coins exorsexism, another load-bearing anti-trans subcategory targeting nonbinary and intersex people.","marker":"[79]"},{"why":"supplies the anti-transmasculinity/transandrophobia framing used in the taxonomy.","marker":"[64]"}],"fun_headline_variants":["Anti-trans TikTok actors reply to pro-trans users 5-to-1","Disassortative TikTok networks reveal anti-trans targeting","Anti-trans TikTok users initiate 5-to-1 reply ratio against pro-trans creators","5-to-1: Anti-trans TikTok replies target pro-trans users"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the classifier's labels for the unannotated network are accurate, but the reported accuracy may be inflated because the same annotated set is used to build the retrieval database, select the prompt, and evaluate the model, with no held-out test split.","fun_headline_variants_meta":{"raw":{"variants":["Anti-trans TikTok actors reply to pro-trans users 5-to-1","Disassortative TikTok networks reveal anti-trans targeting","Anti-trans TikTok users initiate 5-to-1 reply ratio against pro-trans creators","5-to-1: Anti-trans TikTok replies target pro-trans users"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001278,"raw_usage":{"total_tokens":5238,"prompt_tokens":969,"completion_tokens":4269,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":4193}},"tokens_in":585,"tokens_out":4269,"duration_ms":29000,"temperature":1.0,"reasoning_tokens":4193,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:48:09.307185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a strict train/test evaluation: split the 300 annotated videos into disjoint training and test sets, build the RAG retrieval database only from the training set, select the best prompt on the training set, report accuracy on the test set, then recompute the reply-network assortativity and the anti-to-pro interaction ratios using the corrected node labels. If test-set accuracy drops materially below 0.67 or the network statistics lose their strong negative assortativity, the targeting conclusion does not survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"documents the TikTok Research API used for data collection."},{"cited_title":"GET-Tok: A GenAI-Enriched Multimodal TikTok Dataset Documenting the 2022 Attempted Coup in Peru","cited_arxiv_id":"2402.05882","evidence_quote":"documents the video-download failures (private videos, photo collages, Pyktok failures) that shape the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines transmisogyny, a load-bearing anti-trans subcategory in the taxonomy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"coins exorsexism, another load-bearing anti-trans subcategory targeting nonbinary and intersex people."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the anti-transmasculinity/transandrophobia framing used in the taxonomy."}],"review_version":1}