Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

PolitiSky24: U.S. Political Bluesky Dataset with User Stance Labels

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PolitiSky24 gives researchers the first user-level stance labels for the 2024 U.S. election from Bluesky.

desk verdict Useful first Bluesky user-stance resource, but the 81% accuracy is measured on a hashtag-sampled validation set and should be treated as an upper bound on full-dataset label quality. read the letter →

arxiv 2506.07606 v1 pith:EPKZUJ77 submitted 2025-06-09 cs.CL cs.AIcs.IRcs.SI

Peyman Rostami , Vahid Rahimzadeh , Ali Adibi , Azadeh Shakery This is my paper
classification cs.CLcs.AIcs.IRcs.SI
keywords stancedetectionuser-levelBluesky2024U.S.presidentialelectionLLMannotationsocialnetworkdatasetKamalaHarrisDonaldTrump
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces PolitiSky24, a dataset of 16,044 user-target stance pairs built from Bluesky posts about Kamala Harris and Donald Trump in the 2024 U.S. presidential election. It is, to the authors' knowledge, the first user-level stance dataset for that election and the first large stance resource from Bluesky. The dataset labels each user's overall stance toward each candidate as Favor, Against, or Neither, and attaches the reasoning, source posts, and text spans behind each label. The authors also report that their retrieval-plus-LLM labeling pipeline reaches 81% accuracy on a hand-labeled validation set. If the labels hold up, the resource gives researchers an open, explainable basis for studying political alignment on a decentralized social network.

What carries the argument

The load-bearing mechanism is a two-stage label pipeline. In stage one, a stance-based document retriever (KaLM-mini-v1.5, chosen by comparing embedding models on a 700-query relevance set) selects the top posts for and against each target from a user's history; in stage two, an LLM (DeepSeek-Chat-v3, chosen by comparing LLMs on a 446-user human-labeled set) reads those posts and outputs Favor, Against, or Neither, plus the reasoning and the specific spans that support it. The validation set itself is the second mechanism: users are selected by hashtags such as #NeverTrump and #FireKamala, so the accuracy estimate is anchored to users who make their stances explicit.

What would settle it

Sample a set of users who never used any stance-indicative hashtag, have three annotators label their stances under the paper's own guidelines, and run the pipeline on them; if agreement drops well below 81%, the reported accuracy does not transfer to the full dataset.

Watch

Extended reading notes

Core claim

The central claim is that user-level stance can be assigned reliably at scale by combining a stance-aware retriever with a large language model. Using 8,467 users and about 2.85 million English posts collected from political feeds, the pipeline first retrieves the ten most stance-relevant posts for each user-target pair with an embedding model, then prompts DeepSeek-Chat-v3 to label the user's stance toward Harris and Trump with a rationale. On 446 hashtag-sampled users whose stances were independently labeled by human experts, the model reaches 81.2% overall accuracy, with macro F1 of 66.2% for Trump and 72.6% for Harris. The paper also documents where the model fails: most errors come from faulty reasoning, genuinely ambiguous posts, and insufficient retrieved context, and the largest class-level confusion is labeling 'Neither' users as 'Favor' when they neutrally report positive content. The released dataset of 16,044 user-target stance pairs is presented as the first such resource for the 2024 election and as an open, explainable basis for studying political alignment on Bluesky.

Load-bearing premise

The 81.2% accuracy measured on validation users who deliberately used stance hashtags is assumed to hold for all other users, whose labels are produced by the same LLM without human checking.

Editorial extensions

If this is right

  • Researchers can study political stance on Bluesky at user level, using labels accompanied by rationales and source spans rather than bare classifications.
  • The like and repost networks (869,367 and 498,084 edges) make it possible to connect stance labels to community structure and influence.
  • The pipeline provides a reusable recipe for building stance datasets on platforms that, unlike Twitter/X, still offer open API access.
  • The label distribution—majority Against Trump, majority non-opposed to Harris—quantifies the platform's left-leaning political tilt in the 2024 cycle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: because the validation users were selected from users of stance-indicative hashtags, the reported 81.2% accuracy is likely an upper bound for the full dataset, where many users never make their stance explicit.
  • Our inference: a direct test of the transfer assumption would be to run the pipeline on hashtag-free users and compare its labels to fresh human annotations; the paper does not report this comparison.
  • Our inference: the model's tendency to read neutral positive reporting as support means downstream analyses should interpret labels as expressed stance in the users' posts, not as private voting intention.
  • Our inference: because each label carries the LLM's reasoning and quoted spans, the dataset supports post hoc audit and iterative relabeling, so the accuracy could be improved without recollecting data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents PolitiSky24, a user-level stance detection dataset for the 2024 U.S. presidential election, collected from Bluesky. The dataset contains 16,044 user-target stance pairs for Kamala Harris and Donald Trump, enriched with engagement metadata, interaction graphs, and user posting histories. The authors propose a two-stage labeling pipeline: first, an embedding model retrieves stance-relevant posts from a user's history; second, a large language model assigns a stance label (Favor, Against, Neither) with supporting rationale and text spans. The pipeline is configured and evaluated on a hashtag-sampled validation set of 446 users, where DeepSeek-Chat-v3 achieves 81.2% overall accuracy. The authors then apply this pipeline to the full dataset of 8,022 users without human verification of the resulting labels. The paper also analyzes error categories, reports stance distributions, and provides the dataset openly.

Significance. If the labeling quality is reliable, PolitiSky24 fills a clear gap: it is, to the authors' knowledge, the first user-level stance dataset for the 2024 U.S. election, and one of the first stance resources from Bluesky. The dataset's richness—posting histories, interaction networks, rationales, and text spans—could support multiple downstream studies in computational social science and NLP. The paper also contributes a systematic comparison of embedding models and LLMs for stance detection, with transparent prompts and error analysis. However, the central claim of high labeling accuracy (81%) is only validated on a non-representative, hashtag-sampled subset, and the full-dataset labels are produced by an LLM without human checks. Because researchers will likely treat these labels as ground truth, the transferability of the validation accuracy to the full dataset is the load-bearing risk. The authors do acknowledge sampling bias in the Limitations section, but they do not quantify its effect on full-dataset label quality.

major comments (3)
  1. [§4.2.1 and §5.1 (Figure 5)] The reported 81.2% accuracy is computed on a validation set of 446 users sampled via stance-indicative hashtags (Section 4.2.1), but the full dataset comprises 8,022 users who were not subject to any hashtag requirement and whose labels are entirely LLM-generated with no human verification. The paper's own Limitations acknowledges that hashtag users may exhibit 'more extreme or clearly articulated stances,' which makes the validation set easier than the full population. This concern is not merely hypothetical: comparing Table 12 and Table 14, the proportion of Harris 'Neither' labels rises from 39.6% in the human validation set to 52.6% in the full dataset, while the LLM's accuracy on 'Neither' is only 66.8% (Figure 8). The overall accuracy on the full dataset is therefore likely lower than 81.2%. The authors should either provide human-verified labels on a random sample of full-dataset users, or revise the paper's claims to state clearly that 81.2% is a validation-set-specific accuracy and does not necessarily transfer.
  2. [§4.2.3 and Figure 5] The same validation set is used both to select the best-performing LLM and embedding model and to report the final accuracy. This makes the 81.2% figure a best-of-many estimate on that set, which is optimistically biased. A rigorous evaluation would split the human-annotated data into development and test sets (or use nested resampling) so that model selection and final evaluation are independent. In addition, no majority-class baseline or confidence intervals are reported; given the strong label imbalance in the validation set (e.g., Trump 'Against' constitutes 87.6% of human labels), overall accuracy alone is not an informative metric. The authors should report per-class precision, recall, and F1, and compare against a majority-class baseline.
  3. [Abstract and §5.1] The abstract and conclusion state that 'the labeling approach achieves 81% accuracy' without the crucial caveat that this accuracy is measured only on the hashtag-sampled validation set, while the labels in the released dataset are entirely LLM-generated and unverified. This overstates the reliability of the resource. If the authors cannot obtain a representative human-annotated sample, they should at least present the accuracy as an upper bound and explicitly warn downstream users about the potential distribution shift between the validation set and the full dataset, as exemplified by the Harris 'Neither' discrepancy.
minor comments (5)
  1. [§4.2.2] The text says 350 post–target entity pairs were annotated, leading to 4 × 175 = 700 query–stance relevance pairs, but the relationship between 350 and 175 is not explained; please clarify how the 175 pairs are derived from the 350 post–target pairs.
  2. [Table 3] The hashtag '#V oteHarrisWalz202' appears to contain a space; this is likely a typo for '#VoteHarrisWalz202'.
  3. [§5.1] The model name 'LLAMA-3.3-70B-Instruct' should be written as 'Llama-3.3-70B-Instruct' for consistency with the cited technical report.
  4. [Table 12 and Figure 5] The denominator used to compute the 81.2% accuracy is not explicit; the table reports counts of 'Common Users' for each target, but it is unclear whether accuracy is averaged over both targets or computed on the union of user–target pairs. Please state this precisely.
  5. [§5.2] The phrase 'most users do not express opposition toward Harris' is vague; consider reporting the exact percentages from the full dataset distribution rather than qualitative statements, since the dataset's own distribution is central to the paper's findings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: accuracy is measured against independent expert labels; remaining caveats are evaluation-bias concerns, not derivation circularity.

full rationale

PolitiSky24's core claim—that the retrieval-plus-LLM labeling pipeline reaches 81% accuracy—is supported by a comparison of LLM predictions to human expert labels on a held-out validation set (Sections 4.2.3 and 5.1, Figure 5). The human labels are external to the pipeline, so the accuracy is a measured agreement, not a quantity defined by the pipeline's own outputs. The later full-dataset labels are generated by the same pipeline without human verification, and the validation set was hashtag-sampled; the paper's Limitations acknowledges that this may bias toward users with more extreme or clearly articulated stances. Additionally, the same validation set is used both to select the best embedding model and LLM and to report the final accuracy, which makes 81.2% a selection-optimized, in-sample estimate rather than an unbiased forecast for the full dataset. These are important evaluation and generalizability caveats, but they are not circular derivation: no step reduces by construction to its inputs, no load-bearing self-citation is used, and the pipeline is grounded in independent human annotations. Accordingly, no circularity steps are identified.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several hand-set dataset construction choices and domain assumptions about how hashtags, feeds, and LLM outputs map to true user stance. These are not mathematical axioms, but they are the unverified premises that the dataset's validity depends on.

free parameters (5)
  • Minimum posts threshold for target user inclusion (10 posts/reposts) = 10
    Users with at least 10 distinct posts in the three monitored feeds over Nov 12-27, 2024 were defined as target users; this hand-set threshold shapes the sample toward highly active accounts.
  • Context posts per user-target pair (top 5 stance-relevant support + top 5 against = 10 posts) = 10 (5+5)
    The LLM prompt concatenates the top five supporting and top five opposing posts retrieved by the embedding model; number is chosen by hand and directly affects label accuracy.
  • Maximum recent English posts retrieved per user = 1,000
    Up to 1,000 English-language posts per user were retrieved for stance labeling; truncation affects coverage for heavy posters.
  • Validation set size as 5% hashtag-sampled users = 446 users
    5% of users were selected for validation via hashtag-based random sampling; the reported 81% accuracy applies to this hashtag-enriched subset, not a random sample of the full dataset.
  • Feed collection window = Nov 12-27, 2024 (16 days)
    Users were selected from posts in three political feeds during a 16-day period shortly after the election; window choice determines which users enter the dataset.
assumptions (5)
  • domain assumption Users who use stance-indicative hashtags are a valid proxy for identifying validation users with known stances.
    Validation set construction (Sec 4.2.1) selects users who have used expert-curated hashtags such as #NeverTrump or #VoteHarrisWalz202; this assumes hashtag use reliably signals stance and that such users represent the broader population.
  • domain assumption The three selected Bluesky feeds represent U.S. political discourse on Bluesky.
    All target users were drawn from three feeds listed in Appendix D.1.1; if these feeds are ideologically skewed, the dataset inherits that skew, and the paper notes Bluesky is left-leaning.
  • domain assumption LLM-generated stance labels on the full dataset are treated as ground-truth user stances without human verification.
    After validation, DeepSeek-Chat-v3 labels all non-validation users (Sec 4.2.4); the dataset presents these labels as the stance resource, though no sample of full-dataset labels is human-checked.
  • domain assumption Human expert labels on the validation set are reliable.
    The annotation process (Sec 4.2.3) relies on three experts; the paper mentions inter-rater agreement but reports no numeric agreement, so reliability of the gold labels is assumed.
  • domain assumption Embedding retrieval precision at top k is sufficient to provide the LLM with stance-relevant context.
    The pipeline assumes the selected embedding model retrieves the most stance-relevant posts among each user's history, an assumption the error analysis (21% context errors) partially contradicts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PolitiSky24: U.S. Political Bluesky Dataset with User Stance Labels." pith.science (2026). https://pith.science/paper/EPKZUJ77

@misc{pith2026250607606,
  author       = {Pith},
  title        = {Pith review of: PolitiSky24: U.S. Political Bluesky Dataset with User Stance Labels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPKZUJ77}},
  note         = {Machine review of arXiv:2506.07606}
}
read the original abstract

Stance detection identifies the viewpoint expressed in text toward a specific target, such as a political figure. While previous datasets have focused primarily on tweet-level stances from established platforms, user-level stance resources, especially on emerging platforms like Bluesky remain scarce. User-level stance detection provides a more holistic view by considering a user's complete posting history rather than isolated posts. We present the first stance detection dataset for the 2024 U.S. presidential election, collected from Bluesky and centered on Kamala Harris and Donald Trump. The dataset comprises 16,044 user-target stance pairs enriched with engagement metadata, interaction graphs, and user posting histories. PolitiSky24 was created using a carefully evaluated pipeline combining advanced information retrieval and large language models, which generates stance labels with supporting rationales and text spans for transparency. The labeling approach achieves 81\% accuracy with scalable LLMs. This resource addresses gaps in political stance analysis through its timeliness, open-data nature, and user-level perspective. The dataset is available at https://doi.org/10.5281/zenodo.15616911

Figures

Figures reproduced from arXiv: 2506.07606 by the authors.

Figure 1
Figure 1. An illustrative example of the user stance with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Hashtag cloud generated from users’ posts. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. LLM-based annotation pipeline for labeling Bluesky users’ stances toward Trump and Harris. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Performance comparison of various text embedding models for stance-based document retrieval. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of different language models on stance detection task. The plot shows overall [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Distribution of final users’ stances, labeled by [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Confusion matrix showing the performance [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The prompt used for stance labeling using LLMs in the last stage of our stance labeling pipeline. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 11
Figure 11. Figure 11: Active user distribution over time. ‘Active’ [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Word count distribution of the Bluesky posts. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Distribution of validation users’ stances, la [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Quantifying Political Partisanship for Cross-Platform Analyses

    cs.SI 2026-07 reject novelty 5.0 of 10

    Partisanship of individual posts can be scored on a common embedding axis anchored by AllSides news-bias labels, yielding cross-platform scores that transfer from Bluesky/Truth Social to X.

Reference graph

Works this paper leans on

16 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [3]

    In Proceedings of the 2024 ACM on Internet Measurement Conference , pages 76–91, New York, NY , USA

    Looking AT the blue skies of bluesky. In Proceedings of the 2024 ACM on Internet Measurement Conference , pages 76–91, New York, NY , USA. ACM. Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier

    work page 2024

  2. [4]

    arXiv preprint arXiv:2407.21783

    The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Andrea Failla and Giulio Rossetti

    arXiv

  3. [7]

    In Proceedings of the ACM Conext-2024 Workshop on the Decentralization of the Internet , pages 1–7, New York, NY , USA

    Bluesky and the AT protocol: Usable decentralized social media. In Proceedings of the ACM Conext-2024 Workshop on the Decentralization of the Internet , pages 1–7, New York, NY , USA. ACM. Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Ja- yaraman Nair, Diana Inkpen, and Cornelia Caragea. 2021a. P-stance: A large dataset for stance detection in political...

    work page 2024

  4. [8]

    arXiv preprint arXiv:2412.19437

    Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry

    arXiv

  5. [9]

    Bluesky: Network Topology, Polarization, and Algorithmic Curation

    Bluesky: Network topology, polarisation, and algorithmic cu- ration. arXiv preprint arXiv:2405.17571. Dorian Quelle and Alexandre Bovet

    work page Pith review arXiv

  6. [10]

    Preprint, arXiv:2505.06184

    From millions of tweets to actionable insights: Leveraging llms for user profiling. Preprint, arXiv:2505.06184. Majid Zarharan, Maryam Hashemi, Malika Behroozrazegh, Sauleh Eetemadi, Mohammad Taher Pilehvar, and Jennifer Foster

    arXiv

  7. [11]

    FarExStance: Explainable Stance Detection for Farsi

    Farexstance: Explainable stance detection for farsi. arXiv preprint arXiv:2412.14008. Bowen Zhang, Genan Dai, Fuqiang Niu, Nan Yin, Xiao- mao Fan, and Hu Huang

    work page Pith review arXiv

  8. [12]

    arXiv preprint arXiv:2409.15690

    A survey of stance detection on social media: New directions and per- spectives. arXiv preprint arXiv:2409.15690. Chenye Zhao and Cornelia Caragea

    arXiv

Show all 16 references
  1. [13]

    Politics, aged 26, 27, and 32, who provided specialized knowledge essential for accurate stance labeling

    Our annotation process was supported by three male experts in U.S. Politics, aged 26, 27, and 32, who provided specialized knowledge essential for accurate stance labeling. Stance label Description Favor • Directly expressing support for the tar- get entity. • Expressing suppo...

    2024

  2. [14]

    D.1.2 User Post History Users’ post histories are stored using the fields specified in Table 6, which lists each field along with its valid values and detailed descriptions

    In total, 8,561 unique users—each with at least 10 posts across these three feeds—were included in our dataset. D.1.2 User Post History Users’ post histories are stored using the fields specified in Table 6, which lists each field along with its valid values and detailed descr...

    2024

  3. [15]

    Table 12 reports statistics on the number of users labeled by humans, the LLM, and both, categorized by stance and target

    D.2.2 User Stance (Validation dataset) Table 11 presents the schema of the validation user stance dataset. Table 12 reports statistics on the number of users labeled by humans, the LLM, and both, categorized by stance and target. Figure 13 shows the distribution of labels to- ...

    2024

  4. [16]

    Additionally, Table 14 provides statistics on the context posts and LLM- generated fields across the entire dataset

    This table details all dataset fields, including their possible values and descriptions. Additionally, Table 14 provides statistics on the context posts and LLM- generated fields across the entire dataset. E LLM Usage Statement We acknowledge the use of Large Language Mod- els...

    2022

  5. [2020]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8913– 8931, Online

    Zero- Shot Stance Detection: A Dataset and Model using Generalized Topic Representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8913– 8931, Online. Association for Computational Lin- guistics. Sunil Arya, David...

    2020

  6. [2021]

    In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4725–4735, Online

    Knowledge enhanced masked language model for stance detection. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4725–4735, Online. As- sociation for Computational Ling...

    2021

  7. [2024]

    arXiv preprint arXiv:2412.08905

    Phi-4 technical re- port. arXiv preprint arXiv:2412.08905. Emily Allaway and Kathleen McKeown

    arXiv

  8. [2025]

    arXiv preprint arXiv:2501.01028

    Kalm- embedding: Superior training data brings a stronger embedding model. arXiv preprint arXiv:2501.01028. Ujun Jeong, Bohan Jiang, Zhen Tan, H Russell Bernard, and Huan Liu

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.