{"id":"f2d3211e-59ce-40a6-84e1-00663cec496c","arxiv_id":"2505.02250","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A curated dataset of 43,040 TikTok videos, 577,071 comments, and metadata related to eating disorders, with descriptive analyses of engagement, topics, and emotions.","lead":"The authors built and released a dataset of 43,040 TikTok videos tagged with eating-disorder keywords, collected from January 2019 to June 2024, together with nearly 577,000 comments. The dataset lets researchers study how eating-disorder content spreads, how users engage with it, and how the COVID-19 pandemic affected it, with implications for content moderation and mental health.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The curation pipeline's recall is unmeasured: Gemini classifies only text descriptions, so videos with generic captions but ED-relevant visuals/audio are silently dropped, biasing the dataset.","rationale":"The reader identified the Gemini classifier validation as the weakest assumption; our analysis agrees that the curation pipeline's filtering step is the load-bearing element. However, we sharpen the concern: the issue is not only the small validation sample or the unpinned model, but the fact that classification uses only text descriptions, creating an unmeasured recall bias that undermines the 'multimodal' and 'comprehensive' claims. The post-filter precision check (300/300) is good evidence that kept videos are relevant, but it says nothing about relevant videos that were discarded. A biased sample would still distort temporal, engagement, and thematic analyses, which are core to the paper's asserted research value. We keep the verdict at CONDITIONAL (represented as UNCHANGED) because the dataset remains potentially useful, but the required conditions should include a recall audit and full disclosure of the curation parameters.","tokens_in":10439,"tokens_out":5208,"duration_ms":66999,"concrete_test":"Sample 400 videos from the excluded set (the videos removed by the Gemini classifier) and have two independent annotators label each as ED-related or not. Estimate the false-negative rate with a confidence interval; if the lower bound exceeds 10%, the dataset is materially biased. Complement this by re-running the Table 5 prompt on the original 200-video sample with a pinned Gemini model version and reporting per-class precision and recall, not just aggregate accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a curated multimodal dataset of eating-disorder TikTok content. The curation pipeline (Data Filtering; Figure 1) applies the Gemini classifier to video descriptions only (Table 5 prompt), after a keyword-removal step. This is the load-bearing step: the dataset's multimodality is a stated strength, but the classifier never sees the video frames or audio track. In practice, many ED-related TikTok posts have minimal or generic captions (e.g., 'day 1', emoji, 'part 2') while the ED-relevant content is visual or spoken. The 200-video validation and the 300-video post-filter check measure precision on the kept set, not recall on the excluded set. The paper never reports how many relevant videos were discarded by the classifier. If the false-negative rate is substantial, the final 43,040 videos are not a 'comprehensive view' of ED content; they are biased toward textually self-describing posts. This directly threatens the claimed utility for temporal trend analysis and moderation studies. Additionally, the Gemini model version and execution date are not pinned, so the curation cannot be reproduced or audited, and the manually authored keyword-removal list is undisclosed. These factors together make the central dataset claim insecure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents EDTok, a curated dataset of 43,040 TikTok videos related to eating disorders, collected via a set of hashtags and keywords in Table 1, spanning January 2019 to June 2024, alongside 577,071 comments and replies. The curation pipeline (Section 'Data Filtering') first removes videos with manually identified irrelevant keywords, then applies a Google Gemini text classifier to each video description, validating on a random sample of 200 videos (reporting ~99% accuracy) and on 300 post-filter videos (all confirmed relevant). The paper then provides descriptive analyses of engagement statistics, temporal trends, topic modeling of video descriptions and comments, emotion analysis, and a multimodal analysis of a 500-video subsample. The central claim is that EDTok is a comprehensive, multimodal resource for studying eating disorder content on TikTok, enabling analyses of content spread, moderation, engagement, and pandemic-related trends.","tokens_in":10621,"tokens_out":3788,"duration_ms":43861,"significance":"If the dataset's curation is valid, EDTok would be a valuable public resource for the study of eating disorder discourse on TikTok, combining video files, metadata, comments, and temporal coverage across the COVID-19 pandemic. The paper also contributes a reproducible metadata collection framework built on the TikTok Research API, and it is transparent about several limitations. However, the dataset's central validity claim rests on a single classifier applied to text descriptions, with unmeasured recall and incomplete reproducibility details; if those issues are addressed, the dataset could support meaningful research in computational social science and digital health. The descriptive analyses (engagement, temporal, topic, emotion) are straightforward and mostly illustrative, but they do not independently validate the curation.","major_comments":[{"comment":"The curation pipeline applies the Gemini classifier to video descriptions only (Table 5), yet the abstract and Discussion claim a multimodal dataset and a 'comprehensive view' of eating disorder content. Because recall on the excluded set is never measured, videos whose eating-disorder relevance is carried by visuals or audio rather than by text (for example, posts captioned 'day 1' or with emoji-only descriptions) are silently discarded, biasing the final 43,040 videos toward textually self-describing posts. Please estimate the false-negative rate by sampling videos excluded by the classifier and annotating them for eating-disorder relevance, and report this estimate alongside the precision figures. Without a recall estimate, the temporal trend and moderation analyses in Figures 2 and 3 cannot be interpreted as describing the full population of eating-disorder videos on TikTok.","section":"Data Filtering"},{"comment":"Neither the Gemini model version nor the date of execution is provided, and the manual keyword/hashtag removal list that reduced the dataset from 56,472 to 43,040 videos is not disclosed. This makes the curation pipeline non-reproducible and non-auditable, which is a load-bearing limitation for a dataset paper. Please specify the exact model identifier and query date, and include the full exclusion keyword list in an appendix or repository file.","section":"Data Filtering"},{"comment":"The validation of the classifier is based on 200 hand-checked examples with a self-reported 99% accuracy and a 300-video post-filter check with all confirmed related, but no inter-annotator agreement, confidence intervals, or stratified sampling are reported. The prompt in Table 5 explicitly lists terms such as 'anorexia', 'dieting', and 'weight loss', so the 99% figure may not generalize to the hard cases (misspelled hashtags, oblique references, recovery jargon) that motivated the study. I recommend adding a second annotator for a subset of the validation samples, reporting Cohen's kappa, and providing stratified validation across time periods and hashtag families.","section":"Data Filtering"}],"minor_comments":[{"comment":"The dataset flowchart is central to understanding the pipeline, but the exact counts at each stage (e.g., how many videos were removed by the keyword step versus the Gemini step) are not given in the text; please add these numbers so readers can audit the 56,472-to-43,040 reduction.","section":"Figure 1"},{"comment":"The word 'residental' is misspelled and should be 'residential'; the top-words column would also benefit from consistent punctuation (e.g., listing each word separated by commas).","section":"Table 3"},{"comment":"The 'Fighting Battle' topic lists 'fighting, battle and fighter', which reads as a phrase containing 'and' rather than a list of keywords; please use a consistent separator (e.g., 'fighting, battle, fighter').","section":"Table 4"},{"comment":"The paragraph describing the Challenges topic contains 'signficant' instead of 'significant'; please correct the typo.","section":"Text Analysis"},{"comment":"The BERTopic results are presented without any quantitative quality metrics (e.g., topic coherence or topic diversity); adding such measures would strengthen confidence in the interpretability of the topics.","section":"Text Analysis"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a dataset description rather than a hypothesis-driven study; the bar for acceptance should be whether the resource is sufficiently validated and reproducible for the claimed purposes. The missing exclusion list and unmeasured classifier recall are the main correctness risks; the paper's own limitations paragraph acknowledges some but not all of these issues. The reliance on the authors' prior keyword lists is not circular and is acceptable. If the authors can supply a recall estimate, model version, and the exclusion list, the dataset could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Colleague],\n\nQuick take on EDTok: the dataset is a real expansion of the prior 'Hidden in Plain Sight' work—43k videos, 577k comments/replies, five years of metadata, plus a collection framework. That alone is useful for anyone studying ED discourse on TikTok, and the descriptive analyses (temporal, hashtags, topics, emotions) are competently done. The paper is honest about some limitations in the Discussion.\n\nThe soft spot is the curation filter. The Gemini classifier runs on video descriptions only. The stress-test note is onto something: videos with generic captions ('day 1', emoji) but ED-relevant visuals or audio will be dropped, and the paper never measures recall. The 200-video validation and 300-video post-check measure precision on the kept set, not how many relevant videos were discarded. So the 43,040 videos are not a 'comprehensive view'—they are biased toward textually self-describing posts. That matters for temporal trend claims and moderation studies.\n\nAlso, the Gemini model version/date isn't pinned, and the removed keyword list is undisclosed. Those are fixable in a revision. The self-reported 99% accuracy from 200 hand-checked examples with no IAA or CIs is thin, but it's a validation sample issue, not a smoking gun.\n\nI don't agree with the more aggressive version of the stress-test concern that the central claim is 'insecure.' The data is still largely usable for text-based analyses and engagement patterns, as long as the bias is stated. The paper acknowledges the keyword limitation but not the text-only classifier limitation.\n\nSo: worth a serious referee. The authors should disclose the removed keywords, pin the Gemini model, report a proper filter evaluation with recall (e.g., on a sample of discarded videos), and soften the 'comprehensive' language. If those revisions happen, this is a citeable community resource.","headline":"Useful dataset, but the text-only Gemini filter keeps 'comprehensive' from being true; send to review with revisions.","tokens_in":11169,"tokens_out":1753,"would_cite":true,"duration_ms":18620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents EDTok, a curated dataset of 43,040 TikTok videos related to eating disorders, spanning January 2019 to June 2024, together with 577,071 comments and replies and a collection framework.","keywords":["EDTok","eating disorders","TikTok","social media dataset","content moderation","COVID-19 pandemic","topic modeling","multimodal analysis"],"falsifier":"Have two independent human annotators review a random sample of 1,000 videos labeled relevant by the pipeline, drawn across all collection years, and compare their labels to the classifier's; if the human-classifier agreement falls well below 99%, especially on videos without explicit eating-disorder keywords, the filter does not generalize as claimed.","tokens_in":10224,"feed_emoji":"🎥","tokens_out":6983,"duration_ms":76756,"temperature":0.7,"pith_summary":"The paper's aim is to give researchers a large, multimodal public dataset for studying how eating disorder content moves through TikTok. It reports collecting 43,040 videos posted between January 2019 and June 2024, along with their metadata, audio transcripts, 577,071 comments and replies, and the video files, using eating-disorder keywords and hashtags. The authors argue the dataset fills a gap left by text-only or self-report studies, because it captures both the visual and textual dimensions of eating disorder discourse and spans the COVID-19 pandemic. If the collection is as reliable as claimed, it would let researchers track content spread, measure engagement, evaluate moderation evasion such as misspelled hashtags like #edr3c0very, and test whether the pandemic changed the volume and tone of such content. This matters for informing content moderation policy and mental-health interventions on a platform dominated by adolescents.","feed_headline":"A dataset of 43,040 TikTok videos tracks eating disorder content","feed_subtitle":"2019-2024 videos plus comments and engagement data open new research on spread and moderation","key_machinery":"The load-bearing object is the EDTok dataset and its curation pipeline. The pipeline starts with a manually assembled list of eating-disorder keywords and hashtags, including deliberate misspellings such as 'edrec0very' used to evade moderation, queried through TikTok's Research API; the PykTok module downloads the video files; then a two-stage filter removes unrelated posts, first by dropping videos whose metadata matches irrelevant keywords, then by running a large language model prompt on each video description that labels the video as eating-disorder-related or not. The classifier was validated on 200 randomly sampled videos with reported 99% accuracy and then applied to the full collection, followed by a 300-video manual check of the filtered result. This mechanism carries the argument because the dataset's research value depends entirely on the filter separating eating-disorder content from the roughly 20% unrelated videos that the initial keyword query pulls in.","core_discovery":"The central discovery is the dataset itself: a curated corpus of 43,040 TikTok videos on eating-disorder topics, plus 577,071 comments and replies, collected from a query set of keywords and hashtags and filtered through a two-stage pipeline that first removes obviously unrelated posts and then uses an off-the-shelf large language model classifier on video descriptions to keep only eating-disorder-relevant content. The paper reports that a manual review of 200 videos found the classifier about 99% accurate, and that a later random check of 300 videos in the final set found all of them relevant. The dataset spans January 1, 2019 to June 28, 2024, and records engagement totals of about 80 million likes, 537 million views, 577 thousand comments, and 962 thousand shares from 10,897 unique users. In the authors' framing, the contribution is not a finding about eating disorders so much as an instrument: a reproducible collection method and a public resource of video IDs and metadata that can support research on content spread, user engagement, moderation, and the pandemic's influence on online eating-disorder discourse.","pith_inferences":["Inference: The dataset's dependence on a single untuned classifier means its validity could be checked by re-annotating a stratified sample across years, engagement levels, and hashtag families; a large disagreement with human labels would not overturn the corpus but would narrow the claims that can be drawn from it.","Inference: Because the query relies on known hashtags and manual curation, the dataset is likely biased toward recovery and awareness content and may under-represent communities using newer or more obscure jargon; researchers comparing prevalence over time should treat volume changes cautiously.","Inference: The same collection framework could be pointed at other health conditions or at platform policy changes to create comparable datasets, effectively turning the pipeline into a template for studying moderation dynamics on short-video platforms."],"forward_implications":["Researchers can trace monthly video volume and engagement before, during, and after the pandemic, testing whether COVID-19 increased eating-disorder content on TikTok.","The dataset's combination of video files, descriptions, and comments enables multimodal studies linking visual content, caption sentiment, and community responses.","The keyword list and filtering framework can be reused or extended to collect other sensitive health topics, and the video IDs permit replication without redistributing copyrighted videos.","Analysis of engagement and moderation-evading hashtags can inform whether TikTok's moderation efforts are curbing harmful content or inadvertently suppressing recovery-oriented posts.","The included comments and replies allow studying how audiences react emotionally to recovery narratives versus harmful content, as the paper's emotion analysis begins to do."],"supporting_citations":[{"why":"Supplies the earlier hashtag set and documents moderation-evading misspellings like #edr3c0very that the query list extends.","marker":"Bickham et al. 2024"},{"why":"Contributes hashtags previously used to study pro-anorexia communities, forming part of the query set.","marker":"Lerman et al. 2023"},{"why":"Establishes that #EDrecovery content is a substantial TikTok genre, motivating the recovery-focused collection.","marker":"Herrick, Hallward, and Duncan 2021"},{"why":"Reports increased pro-ana/pro-mia content on TikTok around the COVID-19 pandemic, motivating the 2019-2024 collection window.","marker":"Jordan et al. 2021"},{"why":"Provides BERTopic, the method used to derive the paper's topic themes from video descriptions and comments.","marker":"Grootendorst 2022"},{"why":"Supplies Demux, the emotion classifier used in the paper's per-topic emotional analysis.","marker":"Chochlakis et al. 2023"}],"fun_headline_variants":["43K TikTok videos on eating disorders, 2019-2024","Eating disorder TikTok dataset: 43K videos, 537M views","New public dataset tracks eating disorder TikTok content","43,040 TikTok videos map eating disorder discourse","TikTok eating disorder research set: 43K videos, 2019-2024"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The curation pipeline assumes that a single text classifier, written with a manually authored prompt and checked on only 200 videos, correctly identifies eating-disorder relevance for every one of the more than 56,000 descriptions it then labels.","fun_headline_variants_meta":{"raw":{"variants":["43K TikTok videos on eating disorders, 2019-2024","Eating disorder TikTok dataset: 43K videos, 537M views","New public dataset tracks eating disorder TikTok content","43,040 TikTok videos map eating disorder discourse","TikTok eating disorder research set: 43K videos, 2019-2024"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1778,"prompt_tokens":930,"completion_tokens":848,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":758}},"tokens_in":546,"tokens_out":848,"duration_ms":10244,"temperature":1.0,"reasoning_tokens":758,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:56:15.695082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent human annotators review a random sample of 1,000 videos labeled relevant by the pipeline, drawn across all collection years, and compare their labels to the classifier's; if the human-classifier agreement falls well below 99%, especially on videos without explicit eating-disorder keywords, the filter does not generalize as claimed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the earlier hashtag set and documents moderation-evading misspellings like #edr3c0very that the query list extends."},{"cited_title":"Radicalized by Thinness: Using a Model of Radicalization to Understand Pro-Anorexia Communities on Twitter","cited_arxiv_id":"2305.11316","evidence_quote":"Contributes hashtags previously used to study pro-anorexia communities, forming part of the query set."},{"cited_title":"This is just how I cope","cited_arxiv_id":null,"evidence_quote":"Establishes that #EDrecovery content is a substantial TikTok genre, motivating the recovery-focused collection."},{"cited_title":"L.; Garc \\' a, M","cited_arxiv_id":null,"evidence_quote":"Reports increased pro-ana/pro-mia content on TikTok around the COVID-19 pandemic, motivating the 2019-2024 collection window."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Demux, the emotion classifier used in the paper's per-topic emotional analysis."}],"review_version":1}