Pith. sign in

REVIEW 5 major objections 4 minor 89 references

Commenotes: Synthesizing Organic Comments to Support Community-Based Fact-Checking

T0 review · 5 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Organic comments made within two hours of a misleading post contain enough debunking substance that a filter-plus-LLM pipeline can synthesize Community-Note-style fact-checks that users rate as helpful as, or more helpful than, human-writte

desk verdict A plausible, well-scoped system paper whose headline numbers rest on a thinly validated GPT-4o labeler; worth refereeing if the authors tighten the measurement. read the letter →

arxiv 2509.11052 v1 pith:R43H7EJO submitted 2025-09-14 cs.HC

Shuning Zhang , Linzhi Wang , Dai Shi , Yuwei Chuai , Jingruo Chen , Yunyi Chen , Yifan Wang , Yating Wang
show 2 more authors
Xin Yi Hewu Li
This is my paper
classification cs.HC
keywords community-basedfact-checkingorganiccommentsLLMsynthesismisinformationCommunityNotesX/Twitterfilter-then-synthesizeuserevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the slowest part of community fact-checking—the gap between a misleading post appearing and a note being written—can be closed using ordinary user comments. Analyzing 2.2 million replies to 1,841 posts on X, it finds corrective comments arrive fast: 99.3% of misleading posts receive debunking comments within two hours, and a median of 36.5 such comments appear before the community note is displayed. It then builds Commenotes, a two-stage pipeline that filters fact-checking comments and asks an LLM to synthesize them into a concise note. In evaluations, notes made from just the first two hours of comments are useful for 85.8% of posts, and in a 144-participant study the best model's notes were preferred over human notes in 70.1% of head-to-head comparisons. If true, this gives platforms a concrete way to deliver corrective context hours or days earlier and to cover posts that never receive a human note.

What carries the argument

The load-bearing mechanism is the two-stage Commenotes pipeline together with the temporal regularity it exploits. Stage one is a filter: a fine-tuned E5-base model, given a post and a reply, labels the reply as fact-checking or not (accuracy 0.8952 on the held-out set). Stage two is a synthesis: an LLM receives up to 300 filtered comments (randomly sampled if more) and is prompted to write a single 280-character Community-Note-style correction that identifies the inaccuracy, provides evidence, and avoids biased language. The enabling empirical regularity is that corrective comments appear overwhelmingly early—99.3% of posts have one within two hours—so the pipeline never waits for the full

What would settle it

Take a random sample of comments the GPT-4o labeler flagged as fact-checks and have independent annotators apply the paper's three criteria (direct refutation, verifiable evidence, synthesized conclusion); if precision is substantially below 0.82 or errors concentrate in evidence-free disagreement, recompute the cumulative proportion of posts with at least one fact-check comment within two hours. A simpler version: re-run the temporal analysis with a stricter label requiring an external link or specific verifiable data point, and see whether 99.3% survives.

Watch

Extended reading notes

Core claim

The central discovery is that the evidence needed for a fact-check already exists in the comment thread before any note is written, and that it can be extracted and compressed automatically. On the dataset studied, fact-checking comments are not rare late arrivals: 73.1% of posts have at least one within the first hour, 99.3% within two hours, and more than half of all fact-checking comments accumulate before the median note creation time of roughly 9.7 hours. Commenotes operationalizes this: a fine-tuned E5 classifier selects fact-checking comments, and an LLM prompted with the post and those comments generates a note of at most 280 characters in neutral language. The generated notes were r

Load-bearing premise

The whole early-availability result rests on GPT-4o's classification of 2.2 million comments as fact-checks, validated on only 150 manually annotated comments with 0.82 overall accuracy and no reported precision, recall, or error analysis; if the labeler over-tags comments that merely disagree, add context, or opine without evidence, the 99.3%-within-two-hours figure and the average of 23.99 fact-check comments per post are inflated.

Editorial extensions

If this is right

  • Community fact-checking latency can drop from a median of 9.7 hours (note creation) or roughly 22 hours (note display) to about two hours by synthesizing from early comments.
  • Posts that never receive a human community note can still get a corrective context note: in the robustness study, 91-92% of synthesized notes for no-note or not-displayed posts were rated at least somewhat helpful.
  • A moderate amount of discussion is sufficient: using the first 60 comments reaches 74.6% useful notes, and 300 comments captures 83.3% of posts within two hours, so synthesis does not require viral threads.
  • Synthesis quality plateaus near the two-hour mark, so waiting longer yields only marginal gains (90.3% at five hours vs 85.8% at two); early intervention is not sacrificing much accuracy.
  • The marginal cost is low enough for deployment—roughly $0.0065-$0.01 per post with the tested models—making a platform-scale early-warning fact-check layer economically plausible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline numbers depend on a model-labeled corpus; if GPT-4o's 'fact-check' definition is looser than the annotation criteria (e.g., counting mere disagreement as correction), the availability statistics could shrink under stricter verification requirements—an editor's caution, not a paper claim.
  • Deployment would face an adversarial game the study does not measure: coordinated comment flooding or prompt-injection comments could steer synthesized notes, so the pipeline would need input provenance or human-in-the-loop verification before being trusted at scale.
  • The user study's participant pool (mostly Chinese, recruited via posters) and English-only materials mean the 70.1% win rate is a comparative signal, not a universal one; replicating with more diverse samples and multilingual posts would test how far preference generalizes.
  • A natural next step the paper leaves implicit: use synthesized notes as pre-drafts for human note writers, turning the two-hour note into a starting point that humans refine rather than a final displayed note—this would inherit human trust while keeping the latency gain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper investigates whether organic user comments on misleading X posts contain debunking information early enough to support community fact-checking, and whether such comments can be synthesized into community-note-style fact-checks. Using a dataset of 2,225,260 replies on 1,841 posts, the authors use GPT-4o to label fact-checking comments, report that 99.3% of posts receive debunking comments within two hours and a mean of 23.99 fact-check comments per post, and build Commenotes, a two-stage pipeline (fine-tuned E5 filter plus LLM synthesis). Technical evaluations report 85.8% useful commenotes within a two-hour window and 89–92% 'helpful'/'somewhat helpful' ratings in a robustness study. A user study (N=144) finds that grok-4-synthesized notes are preferred over human-written Community Notes in 70.1% of pairwise comparisons.

Significance. If the empirical claims hold, this is a meaningful contribution to the community-fact-checking literature: it identifies a new, early source of corrective information and demonstrates a practical pipeline for reducing the latency of community notes. The paper's scope is substantial—over two million comments, four synthesis models, a robustness study on posts without notes, and a comparative user study against human notes. The cost analysis and the emphasis on posts not covered by Community Notes are also valuable. However, the central quantitative claims rest on a labeling methodology whose validation is currently too thin to support the headline numbers, and the evaluation framework does not yet establish that the synthesized notes are factually corrective rather than merely perceived as helpful. The framework itself is plausible and the user-study design is generally sound, but the evidence needs strengthening before the paper's central feasibility claim can be accepted.

major comments (5)
  1. [§3.2 and §3.3.2] The RQ1 headline results—99.3% of posts receiving debunking comments within two hours and a mean of 23.99 fact-check comments per post—are computed entirely from GPT-4o labels. The validation of this labeler is limited to N=150 manually annotated comments with only overall accuracy reported (0.82); no precision, recall, confusion matrix, or per-post error analysis is given. Moreover, the prompt in Appendix C asks only whether the comment 'contains fact-checking information that points out inaccuracies,' which does not encode the three criteria in Appendix A (direct refutation, verifiable evidence/source, synthesized conclusion). If the labeler over-tags comments that merely disagree, give context, or express opinion, the 99.3% and 23.99 numbers are inflated. Please validate the labeler on a larger stratified sample, report per-class precision/recall and error patterns, and either re-esti
  2. [§4.2 and Table 2] The filter stage is fine-tuned and evaluated against the same GPT-4o labels used in §3.2. Table 2 therefore reports agreement with the automatic labeler, not with human ground truth. This means the reported E5 accuracy (0.8952) and recall (0.9068) do not establish that the filter identifies actual fact-checking comments; they only show that the fine-tuned model reproduces GPT-4o's decisions. The downstream synthesis quality depends on the filter's precision against human judgments. Please construct a human-annotated test set (using Appendix A criteria), report filter precision/recall on it, and, if the filter inherits labeler bias, quantify how that bias propagates into the synthesized notes.
  3. [§5.1 and Appendix B] The 'useful' annotation criterion explicitly does not assess the truthfulness of the evidence: Appendix B states that 'the validity or absolute truthfulness of the evidence provided in a note is not assessed in this phase.' Consequently, the 85.8% usefulness result and the user-study helpfulness ratings establish that users find the notes plausible and helpful, but not that the notes are factually correct or that they debunk the post. Since the paper's central claim is about fact-checking, this is a load-bearing gap. Please add a factual-accuracy check on a sample of synthesized notes (e.g., against external sources or expert annotation) and report the accuracy of the final commenotes, or clearly reframe the claims as 'perceived helpfulness' rather than 'fact-checking accuracy.' The Limitations section itself acknowledges the risk of amplifying inaccurate information, which is precisely
  4. [§5.2.2] In the robustness evaluation, 40 of 124 posts are dropped because they have fewer than 25 fact-check comments (3 in the 'with note' condition, 11 in 'without display,' and 26 in 'without note'). The reported helpfulness rates of 89.2%, 91.2%, and 92.0% are computed only on the remaining 84 posts. This is a selection on the dependent variable: the pipeline abstains on a large fraction of posts—especially 26 of 49 'without note' posts—and the reported rates do not reflect end-to-end coverage. Please report the full 124-post denominator, treat abstentions as failures in an end-to-end metric, and discuss the precision-coverage tradeoff. Without this, the robustness claim for posts without notes is overstated.
  5. [§3.2, §4.2, §5.1] The evaluation pipeline has a partially self-referential structure: GPT-4o is used to create the filter labels and is also one of the synthesis models, and the 'usefulness' criterion (a good-faith attempt to provide reasons or evidence, without truth verification) is close to the surface property the synthesis prompt explicitly asks the LLM to produce. This does not make the results invalid, but it means the user-study win rate may partly reflect the models' ability to produce the requested format rather than genuine fact-checking quality. Please discuss this and empirically address it: for example, have independent human annotators apply Appendix A criteria to a sample of synthesized notes, and compare win rates on factually verified versus unverified notes.
minor comments (4)
  1. [§3.3.3 and Table 1] The verified/unverified author statistics are inconsistent: the text reports M=27.76 vs. M=15.95 and proportions 9.69% vs. 12.35%, but Table 1 reports average numbers 144.95 vs. 66.85 and proportions 6.75% vs. 7.42%. Please reconcile.
  2. [§5.2.1 and §7.1] The Discussion (Section 7.1) states that '73.1% of posts received such comments within the first hour,' but 73.1% in Figure 4a is the proportion of useful commenotes synthesized from one hour of comments, not the proportion of posts receiving fact-checking comments. Please correct this conflation.
  3. [§6.1] The material selection says 'failed synthesis attempts were excluded to maintain sample integrity,' but no count of failures is reported. Given the robustness evaluation's abstention rate, please report how many of the 60 selected posts failed synthesis and why.
  4. [Appendix C] The filtering prompt given in Appendix C differs from the annotation criteria in Appendix A: it does not instruct the model to require verifiable evidence or a synthesized conclusion. This may explain labeler disagreement with human annotators and should be aligned or explicitly justified.

Circularity Check

2 steps flagged · score 4.0 of 10

Filter accuracy is measured against the same GPT-4o labels used to train the filter, and the 'usefulness' metric mirrors the synthesis prompt; the user-study comparison to human notes provides partial independent support.

  1. fitted input called prediction [Section 3.2 / Section 5.1 / Section 5.2.1, Table 2]
    "We prompted GPT-4o to identify fact-check comments... GPT-4o achieved an accuracy of 0.82 on this validation set (N=150), indicating fair accuracy. ... We fine-tuned each model for five epochs on the dataset described in Section 3.2, which was split into training (80%), validation (10%), and testing (10%) sets... We report performance on the test set. ... The accuracy of all models exceeded 0.85, while E5-base has the highest accuracy and recall, exceeding or reaching 0.90."

    The small LMs used in the filter stage are trained on GPT-4o's labels and then evaluated on a held-out portion of the same GPT-4o-labeled dataset. The reported 'accuracy' therefore measures agreement with GPT-4o, not with human fact-check labels. The only human validation of GPT-4o is N=150 with an overall accuracy of 0.82 and no reported precision/recall or error analysis. The paper presents this agreement as evidence that the filter can 'accurately complete the filter stage's task,' but the model is fitted to reproduce the labeler and then tested on the same labeler's output, making the benchmark self-referential.

  2. self definitional [Appendix B.1 vs Appendix D / Section 5.1]
    "Your job is to find fact-check information in those comments that indicates that there is something inaccurate in the post and use that information to write a piece of Community Note yourself. ... To be classified as Useful, a note must... 1. Identifies Inaccuracy: The note must explicitly or implicitly point out a factual inaccuracy... 2. Provides Substantiation: The note must substantiate its challenge with reasons, evidence, or relevant information."

    The 'useful' criterion used in the technical evaluation is defined by the same surface properties that the synthesis prompt explicitly instructs the model to produce: identify an inaccuracy and provide substantiating context. Thus the 85.8% 'useful' figure largely measures whether the LLM followed its own instruction, not whether the note is factually corrective. Appendix B explicitly waives assessment of truthfulness ('validity or absolute truthfulness of the evidence provided in a note is not assessed'), so the metric cannot independently validate the paper's fact-checking claim. The evaluation loop is self-definitional: the output is judged useful if it matches the specification it was prompted to satisfy.

full rationale

The paper's central RQ1 result (99.3% of posts receive debunking comments within two hours) is an empirical measurement using GPT-4o as an annotator; this is a validity concern rather than a formal circularity because the labeler is an instrument applied to the data, not a parameter fitted to the outcome. The filter-stage evaluation is more clearly circular: E5 and other small LMs are trained on GPT-4o labels and their 'accuracy' is reported against a test set drawn from the same GPT-4o labeling process. That is an agreement benchmark, not an independent estimate of fact-checking performance, and the small 150-example human validation with only 0.82 accuracy does not break the self-reference. The synthesis-stage 'usefulness' evaluation is also partly self-definitional because the annotators use a criterion essentially identical to the synthesis prompt's instruction, and Appendix B disclaims any check of truthfulness. However, the user study (N=144) comparing commenotes against real human-written Community Notes in a blind, randomized setting provides an external, non-circular benchmark for the relative helpfulness claim, including the 70.1% win rate for grok-4. That independent comparison keeps the central claim from fully reducing to the paper's own definitions. Overall, there are two partial circularities in the technical evaluations, but the headline user-study result retains independent content, so the score is 4 rather than higher.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All central quantities are empirical. The main free choices are the time window, the input cap, and the exclusion threshold, plus the labeling and usefulness definitions. No new physical or formal entities are postulated.

free parameters (4)
  • Two-hour synthesis window = 2 hours
    Selected post hoc from the temporal evaluation (Figure 4a), where 2 hours gives 85.8% useful notes and the 5-hour gain is not significant; not cross-validated on an independent set.
  • 300-comment input threshold = 300
    Chosen to fit LLM context windows and because 83.3% of posts receive fewer than 300 comments in two hours (Section 4.4); not optimized on held-out data.
  • 25 fact-check comment minimum for synthesis = 25
    In the robustness study, posts with fewer than 25 fact-check comments are excluded from generation, dropping 40 of 124 posts (Section 5.2.2). This biases the reported helpfulness toward comment-rich posts.
  • Filtering temperature and top-p = 0.6 and 1.0
    Empirically set for the filtering phase (Section 4.4); affects reproducibility though not the core claim.
assumptions (4)
  • domain assumption The Chuai et al. dataset accurately records post timestamps, note display times, and direct replies on X.
    All temporal statistics in Section 3 depend on the integrity of this externally collected dataset; reply and note ordering is used to separate organic comments from note-contaminated replies.
  • domain assumption GPT-4o fact-check labels generalize from 150 annotated examples to 2.2 million comments.
    Section 3.2 reports only 0.82 accuracy at N=150 and then uses these labels as ground truth for temporal analysis and for training the filter.
  • ad hoc to paper A 'useful' note is defined by good-faith evidence provision, not by verified factual accuracy.
    Appendix B states 'The validity or absolute truthfulness of the evidence provided in a note is not assessed,' which is an assumption linking perceived helpfulness to fact-check quality.
  • domain assumption Replies written before note display are unaffected by the note and reflect organic user responses.
    Section 3.1 excludes later replies to avoid note contamination, but early replies could still include coordinated or otherwise non-organic content.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Commenotes: Synthesizing Organic Comments to Support Community-Based Fact-Checking." pith.science (2026). https://pith.science/paper/R43H7EJO

@misc{pith2026250911052,
  author       = {Pith},
  title        = {Pith review of: Commenotes: Synthesizing Organic Comments to Support Community-Based Fact-Checking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R43H7EJO}},
  note         = {Machine review of arXiv:2509.11052}
}
read the original abstract

Community-based fact-checking is promising to reduce the spread of misleading posts at scale. However, its effectiveness can be undermined by the delays in fact-check delivery. Notably, user-initiated organic comments often contain debunking information and have the potential to help mitigate this limitation. Here, we investigate the feasibility of synthesizing comments to generate timely high-quality fact-checks. To this end, we analyze over 2.2 million replies on X and introduce Commenotes, a two-phase framework that filters and synthesizes comments to facilitate fact-check delivery. Our framework reveals that fact-checking comments appear early and sufficiently: 99.3\% of misleading posts receive debunking comments within the initial two hours since post publication, with synthesized \textit{commenotes} successfully earning user trust for 85.8\% of those posts. Additionally, a user study (N=144) found that the synthesized commenotes were often preferred, with the best-performing model achieving a 70.1\% win rate over human notes and being rated as significantly more helpful.

Figures

Figures reproduced from arXiv: 2509.11052 by the authors.

Figure 1
Figure 1. The algorithm flow of Commenotes, which could be decomposed into two stages: filtering and synthesizing. During filtering, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Average count of fact-checking comments, and (b) percentage of fact-check comments among all comments across [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The change of fact-check comments’ number and their proportion relative to the cumulative total of fact-check comments [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: (a) Performance based on the time window from which comments were collected, measured since the post’s creation. (b) [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Technical evaluation of commenotes’ quality across four LLMs. (a) Distribution of helpfulness ratings for notes generated [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: The helpfulness scores of commenotes across different model settings. (a), (b), (e), (f ) showed the helpfulness scores of the notes [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The winning rate of the commenotes compared with human notes for different models: (a) gpt-4o, (b) grok-4, (c) claude-3.7- sonnet, (d) qwen2.5-72b-instruct. 50%, although neither was statistically different from the baseline. The general decrease in commenotes’ win rat…
Figure 8
Figure 8. Figure 8: The commenotes’ characteristics for different models: (a) gpt-4o, (b) grok-4, (c) claude-3.7-sonnet, (d) qwen2.5-72b-instruct. significant difference suggests that our algorithm’s prompting strategies were effective in guiding the LLMs to produce content that is percei…
Figure 9
Figure 9. Figure 9: The effect of different topics on different aspects of ratings for the commenotes synthesized by grok-4, with error bar showing [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The effect of user demographics on different aspects of ratings for the notes synthesized by the grok-4 model, with errorbar [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: The interfaces used in the user evaluation study. [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]
Figure 12
Figure 12. Figure 12: The effect of topic factors on users’ ratings for (a) overall, (b) claude-3.7-sonnet, (c) gpt-4o, and (d) qwen2.5-72b-instruct [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]
Figure 13
Figure 13. Figure 13: The effect of user demographics (i.e., political stance, and polarization) on different rating metrics for (a) Overall, (b) claude [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 1 canonical work pages

  1. [1]

    Marwan Albahar. 2021. A hybrid model for fake news detection: Leveraging news content and user comments in fake news.IET Information Security15, 2 (2021), 169–177

    2021

  2. [2]

    Jennifer Allen, Antonio A Arechar, Gordon Pennycook, and David G Rand. 2021. Scaling up fact-checking using the wisdom of crowds.Science Advances7, 36 (2021), eabf4393

    2021

  3. [3]

    Abdulrahman Alrumaih, Ali Al-Sabbagh, Ruaa Alsabah, Harith Kharrufa, and James Baldwin. 2020. Sentiment analysis of comments in social media. International Journal of Electrical & Computer Engineering (2088-8708)10, 6 (2020)

    2020

  4. [4]

    Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. 2022. Fine-tuning language models to find agreement among humans with diverse preferences.Advances in Neural Information Processing Systems35 (2022), 38176–38189

    2022

  5. [5]

    Robert M Bond and R Kelly Garrett. 2023. Engagement with fact-checked posts on Reddit.PNAS Nexus2, 3 (2023), pgad018

    2023

  6. [6]

    Gianluca Bonifazi, Francesco Cauteruccio, Enrico Corradini, Michele Marchetti, Giorgio Terracina, Domenico Ursino, and Luca Virgili. 2024. Representation, detection and usage of the content semantics of comments in a social platform.Journal of Information Science50, 2 (2024), 317–341

    2024

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in Neural Information Processing Systems33 (2020), 1877–1901

    2020

  8. [8]

    Yuyan Bu, Qiang Sheng, Juan Cao, Peng Qi, Danding Wang, and Jintao Li. 2023. Combating online misinformation videos: Characterization, detection, and future directions. InProceedings of the 31st ACM International Conference on Multimedia. 8770–8780

    2023

Show all 89 references
  1. [9]

    Jason W Burton, Ezequiel Lopez-Lopez, Shahar Hechtlinger, Zoe Rahwan, Samuel Aeschbach, Michiel A Bakker, Joshua A Becker, Aleks Berditchevskaia, Julian Berger, Levin Brinkmann, et al . 2024. How large language models can reshape collective intelligence.Nature Human Behaviour8...

    2024

  2. [10]

    CBS News. 2025. Meta starts testing community-sourced fact-checking based on X’s Community Notes. https://www.cbsnews.com/sanfrancisco/ news/meta-community-notes-crowd-sourced-fact-checking-based-on-x-twitter-testing/ [Accessed: 2025-09-01]

    2025

  3. [11]

    Mingming Cheng and Xin Jin. 2019. What do Airbnb users care about? An analysis of online review comments.International Journal of Hospitality Management76 (2019), 58–70

    2019

  4. [12]

    Yuwei Chuai, Moritz Pilarski, Thomas Renault, David Restrepo-Amariles, Aurore Troussel-Clément, Gabriele Lenzini, and Nicolas Pröllochs. 2024. Community-based fact-checking reduces the spread of misleading posts on social media.arXiv preprint arXiv:2409.08781(2024)

    2024 arXiv

  5. [13]

    Yuwei Chuai, Anastasia Sergeeva, Gabriele Lenzini, and Nicolas Pröllochs. 2025. Community fact-checks trigger moral outrage in replies to misleading posts on social media. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–23. 22 Commenotes: Synt...

    2025

  6. [14]

    Yuwei Chuai, Haoye Tian, Nicolas Pröllochs, and Gabriele Lenzini. 2024. Did the roll-out of community notes reduce engagement with misinformation on X/Twitter?Proceedings of the ACM on Human-Computer Interaction8, CSCW2 (2024), 1–52

    2024

  7. [15]

    Luke Collins. 2015. How can semantic annotation help us to analyse the discourse of climate change in online user comments?Linguistik Online70, 1 (2015), 43–60

    2015

  8. [16]

    X Corp. 2025. About Community Notes on X. https://help.x.com/en/using-x/community-notes. [Accessed: 2025-09-01]

    2025

  9. [17]

    Abhisek Dash, Anurag Shandilya, Arindam Biswas, Kripabandhu Ghosh, Saptarshi Ghosh, and Abhijnan Chakraborty. 2019. Summarizing user- generated textual content: Motivation and methods for fairness in algorithmic summaries.Proceedings of the ACM on Human-Computer Interaction 3,...

    2019

  10. [18]

    Bakker, Jay Baxter, and Martin Saveski

    Soham De, Michiel A. Bakker, Jay Baxter, and Martin Saveski. 2025. Supernotes: Driving consensus in crowd-sourced fact-checking. InProceedings of the ACM Web Conference 2025. 3751–3761

    2025

  11. [19]

    Chiara Patricia Drolsbach, Kirill Solovev, and Nicolas Pröllochs. 2024. Community notes increase trust in fact-checking on social media.PNAS Nexus3, 7 (2024), pgae217

    2024

  12. [20]

    Ziv Epstein, Gordon Pennycook, and David Rand. 2020. Will the crowd game the algorithm? Using layperson judgments to combat misinformation on social media by downranking distrusted sources. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–11

    2020

  13. [21]

    Sara Fish, Paul Gölz, David C Parkes, Ariel D Procaccia, Gili Rusak, Itai Shapira, and Manuel Wüthrich. 2024. Generative social choice. InProceedings of the 25th ACM Conference on Economics and Computation. 985–985

    2024

  14. [22]

    Daniyal Ganiuly and Assel Smaiyl. 2024. Ethics of using llms in content moderation on twitter.International Journal of Innovative Science and Research Technology (IJISRT)(2024)

    2024

  15. [23]

    Yang Gao, Maggie Mengqing Zhang, and Huaxia Rui. 2024. Can crowdchecking curb misinformation? Evidence from community notes.Evidence from Community Notes (October 17, 2024)(2024)

    2024

  16. [24]

    Spiros V Georgakopoulos, Sotiris K Tasoulis, Aristidis G Vrahatis, and Vassilis P Plagianakos. 2018. Convolutional neural networks for toxic comment classification. InProceedings of the 10th Hellenic Conference on Artificial Intelligence. 1–6

    2018

  17. [25]

    Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking.Transactions of the Association for Computational Linguistics10 (2022), 178–206

    2022

  18. [26]

    Suhaib Kh Hamed, Mohd Juzaiddin Ab Aziz, and Mohd Ridzwan Yaakub. 2023. Fake news detection model on social media by leveraging sentiment analysis of news content and emotion analysis of users’ comments.Sensors23, 4 (2023), 1748

    2023

  19. [27]

    Bing He, Mustaque Ahamad, and Srijan Kumar. 2023. Reinforcement learning-based counter-misinformation response generation: a case study of COVID-19 vaccine misinformation. InProceedings of the ACM Web Conference 2023. 2698–2709

    2023

  20. [28]

    Ming He, Yong Ge, Enhong Chen, Qi Liu, and Xuesong Wang. 2017. Exploring the emerging type of comment for online videos: Danmu.ACM Transactions on the Web (TWEB)12, 1 (2017), 1–33

    2017

  21. [29]

    Benjamin D Horne and Dorit Nevo. 2025. People adhere to content warning labels even when they are wrong due to ecologically rational adaptations. Scientific Reports15, 1 (2025), 13896

    2025

  22. [30]

    Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander van der Linden, and Jon Roozenbeek. 2025. Generative language models exhibit social identity biases.Nature Computational Science5, 1 (2025), 65–75

    2025

  23. [31]

    Shan Jiang and Christo Wilson. 2018. Linguistic signals under misinformation and fact-checking: Evidence from user comments on social media. Proceedings of the ACM on Human-Computer Interaction2, CSCW (2018), 1–23

    2018

  24. [32]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020)

    2020 arXiv

  25. [33]

    Hyunuk Kim and Dylan Walker. 2020. Leveraging volunteer fact checking to identify misinformation about COVID-19 in social media.Harvard Kennedy School Misinformation Review1, 3 (2020)

    2020

  26. [34]

    Eklekta Kristo. 2021. Computer-aided analysis of video comments for requirements analysis. (2021)

    2021

  27. [35]

    Haiwen Li, Soham De, Manon Revel, Andreas Haupt, Brad Miller, Keith Coleman, Jay Baxter, Martin Saveski, and Michiel A Bakker. 2025. Scaling human judgment in community notes with LLMs.arXiv preprint arXiv:2506.24118(2025)

    2025

  28. [36]

    Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024. Automatic and universal prompt injection attacks against large language models.arXiv preprint arXiv:2403.04957(2024)

    2024 arXiv

  29. [37]

    Amy Madden, Ian Ruthven, and David McMenemy. 2013. A classification scheme for content analyses of YouTube video comments.Journal of Documentation69, 5 (2013), 693–714

    2013

  30. [38]

    Robertson, Sara Douglas, Roxanne Raine, and Bryan Semaan

    Misa Maruyama, Scott P. Robertson, Sara Douglas, Roxanne Raine, and Bryan Semaan. 2017. Social watching a civic broadcast: understanding the effects of positive feedback and other users’ opinions. InProceedings of the 2017 ACM Conference on Computer Supported Cooperative Work ...

    2017

  31. [39]

    Meta Transparency Center. 2025. Community Notes on Instagram. https://transparency.meta.com/features/community-notes [Accessed: 2025-09-01]

    2025

  32. [40]

    Nicholas Micallef, Bing He, Srijan Kumar, Mustaque Ahamad, and Nasir Memon. 2020. The role of the crowd in countering misinformation: A case study of the COVID-19 infodemic. In2020 IEEE International Conference on Big Data (big data). IEEE, 748–757

    2020

  33. [41]

    Muaadh Noman, Selin Gurgun, Keith Phalp, and Raian Ali. 2024. Designing social media to foster user engagement in challenging misinformation: a cross-cultural comparison between the UK and Arab countries.Humanities and Social Sciences Communications11, 1 (2024), 1–13. 23 Zhang et al

    2024

  34. [42]

    Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria-Esther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, et al. 2020. Bias in data-driven artificial intelligence systems—An introductory survey.Wiley...

    2020

  35. [43]

    Deokgun Park, Simranjit Sachar, Nicholas Diakopoulos, and Niklas Elmqvist. 2016. Supporting comment moderators in identifying high quality online news comments. InProceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1114–1125

    2016

  36. [44]

    Sotiris Pelekis, Thanos Koutroubas, Afroditi Blika, Anastasis Berdelis, Evangelos Karakolis, Christos Ntanos, Evangelos Spiliotis, and Dimitris Askounis. 2025. Adversarial machine learning: a review of methods, tools, and critical industry sectors.Artificial Intelligence Revie...

    2025

  37. [45]

    Gordon Pennycook and David G Rand. 2019. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences116, 7 (2019), 2521–2526

    2019

  38. [46]

    Jürgen Pfeffer, Daniel Matter, and Anahit Sargsyan. 2023. The half-life of a tweet. InProceedings of the International AAAI Conference on Web and Social Media, Vol. 17. 1163–1167

    2023

  39. [47]

    X Developer Platform. 2025. Community Notes Quickstart. https://docs.x.com/x-api/community-notes/quickstart. [Accessed: 2025-09-01]

    2025

  40. [48]

    Nicolas Pröllochs. 2022. Community-based fact-checking on Twitter’s Birdwatch platform. InProceedings of the International AAAI Conference on Web and Social Media, Vol. 16. 794–805

    2022

  41. [49]

    Peng Qi, Zehong Yan, Wynne Hsu, and Mong Li Lee. 2024. Sniffer: Multimodal large language model for explainable out-of-context misinformation detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13052–13062

    2024

  42. [50]

    Dorian Quelle and Alexandre Bovet. 2024. The perils and promises of fact-checking with large language models.Frontiers in Artificial Intelligence7 (2024), 1341697

    2024

  43. [51]

    Marco Tulio Ribeiro, Tongshuang Wu, and Carlos Guestrin. 2020. Beyond accuracy: behavioral testing of NLP models with checkList. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 4902–4912

    2020

  44. [52]

    Craig T Robertson, Rachel R Mourão, and Esther Thorson. 2020. Who uses fact-checking sites? The impact of demographics, political antecedents, and media use on fact-checking site awareness, attitudes, and behavior.The International Journal of Press/Politics25, 2 (2020), 217–237

    2020

  45. [53]

    Lanyu Shang, Yang Zhang, Daniel Zhang, and Dong Wang. 2020. Fauxward: a graph neural network approach to fauxtography detection using social media comments.Social Network Analysis and Mining10, 1 (2020), 76

    2020

  46. [54]

    Yunhee Shim and Shagun Jhaver. 2024. Incorporating procedural fairness in flag submissions on social media platforms.arXiv preprint arXiv:2409.08498 (2024)

    2024 arXiv

  47. [55]

    Fedor Sizov, Cristina España-Bonet, Josef van Genabith, Roy Xie, and Koel Dutta Chowdhury. 2024. Analysing translation artifacts: A comparative study of LLMs, NMTs, and human translations. InProceedings of the Ninth Conference on Machine Translation. 1183–1199

    2024

  48. [56]

    Jeremy Straub and Matthew Spradling. 2022. Americans’ perspectives on online media warning labels.Behavioral Sciences12, 3 (2022), 59

    2022

  49. [57]

    Tiktok. 2025. Testing a new feature to enhance content on TikTok. https://newsroom.tiktok.com/en-us/footnotes [Accessed: 2025-09-01]

    2025

  50. [58]

    Nathan Walter, John J Brooks, Camille J Saucier, and Sapna Suresh. 2021. Evaluating the impact of attempts to correct health misinformation on social media: A meta-analysis.Health Communication36, 13 (2021), 1776–1784

    2021

  51. [59]

    Haoran Wang and Kai Shu. 2023. Explainable claim verification via knowledge-grounded reasoning with large language models. InFindings of the Association for Computational Linguistics: EMNLP 2023. 6288–6304

    2023

  52. [60]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training.arXiv preprint arXiv:2212.03533(2022)

    2022 arXiv

  53. [61]

    Stephanie Wang, Shengchun Huang, Alvin Zhou, and Danaë Metaxa. 2024. Lower quantity, higher quality: Auditing news content and user perceptions on Twitter/X algorithmic versus chronological timelines.Proceedings of the ACM on Human-Computer Interaction8, CSCW2 (2024), 1–25

    2024

  54. [62]

    Greta Warren, Irina Shklovski, and Isabelle Augenstein. 2025. Show me the work: Fact-checkers’ requirements for explainable automated fact-Checking. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–21

    2025

  55. [63]

    Wikipedia contributors. 2025. Bilibili. https://en.wikipedia.org/wiki/Bilibili [Accessed: 2025-09-01]

    2025

  56. [64]

    Bo Wu, Wen-Huang Cheng, Yongdong Zhang, Qiushi Huang, Jintao Li, and Tao Mei. 2017. Sequential prediction of social media popularity with deep temporal context networks. InProceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI). 3062–3...

    2017 doi

  57. [65]

    Danni Xu, Shaojing Fan, and Mohan Kankanhalli. 2023. Combating misinformation in the era of generative AI models. InProceedings of the 31st ACM International Conference on Multimedia. 9291–9298

    2023

  58. [66]

    Aman Yadav, Rishab Sharma, and Fatemeh H Fard. 2020. A semantic-based framework for analyzing app users’ feedback. In2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 572–576

    2020

  59. [67]

    Yuta Yanagi, Ryohei Orihara, Yuichi Sei, Yasuyuki Tahara, and Akihiko Ohsuga. 2020. Fake news detection with generated comments for news articles. In2020 IEEE 24th International Conference on Intelligent Engineering Systems (INES). IEEE, 85–90

    2020

  60. [68]

    Joshua C Yang, Damian Dalisan, Marcin Korecki, Carina I Hausladen, and Dirk Helbing. 2024. Llm voting: Human choices and ai collective decision-making. InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7. 1696–1708

    2024

  61. [69]

    Youtube Official Blog. 2024. Testing new ways to offer viewers more context and information on videos. https://blog.youtube/news-and-events/new- ways-to-offer-viewers-more-context/ [Accessed: 2025-09-01]. 24 Commenotes: Synthesizing Organic Comments to Support Community-Based ...

    2024

  62. [70]

    Xiaomin Yu, Yezhaohui Wang, Yanfang Chen, Zhen Tao, Dinghao Xi, Shichao Song, Simin Niu, and Zhiyu Li. 2024. Fake artificial intelligence generated contents (faigc): A survey of theories, detection methods, and opportunities.arXiv preprint arXiv:2405.00711(2024)

    2024 arXiv

  63. [71]

    Zhenrui Yue, Huimin Zeng, Yimeng Lu, Lanyu Shang, Yang Zhang, and Dong Wang. 2024. Evidence-driven retrieval augmented response generation for online misinformation. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis...

    2024

  64. [72]

    Fengzhu Zeng and Wei Gao. 2024. Justilm: Few-shot justification generation for explainable fact-checking of real-world claims.Transactions of the Association for Computational Linguistics12 (2024), 334–354

    2024

  65. [73]

    Alice Qian Zhang, Kaitlin Montague, and Shagun Jhaver. 2023. Cleaning up the streets: Understanding motivations, mental models, and concerns of users flagging social media content.arXiv preprint arXiv:2309.06688(2023)

    2023 arXiv

  66. [74]

    Zhang, Lea Verou, and David Karger

    Amy X. Zhang, Lea Verou, and David Karger. 2017. Wikum: Bridging discussion forums and wikis using recursive summarization. InProceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 2082–2096

    2017

  67. [75]

    Daniel Yue Zhang, Lanyu Shang, Biao Geng, Shuyue Lai, Ke Li, Hongmin Zhu, Md Tanvir Amin, and Dong Wang. 2018. Fauxbuster: A content-free fauxtography detector using social media comments. In2018 IEEE International Conference on Big Data (big data). IEEE, 891–900

    2018

  68. [76]

    Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, and Ahmed El-Kishky. 2023. Twhin-bert: A socially-enriched pre-trained language model for multilingual tweet representations at twitter. InProceedings of the 29th ACM SIGKDD Conference on Knowle...

    2023

  69. [77]

    Yongjun Zhang, Hao Lin, Yi Wang, and Xinguang Fan. 2023. Sinophobia was popular in Chinese language communities on Twitter during the early COVID-19 pandemic.Humanities and Social Sciences Communications10, 1 (2023), 1–12

    2023

  70. [78]

    Yusen Zhang, Li Zhou, Harini Suresh, Imed Zitouni, and Milad Shokouhi. 2024. Fair abstractive summarization of diverse perspectives. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies...

    2024

  71. [79]

    Kaitlyn Zhou, Dan Jurafsky, and Tatsunori B Hashimoto. 2023. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 5506–5524

    2023

  72. [80]

    Xinyi Zhou, Ashish Sharma, Amy X Zhang, and Tim Althoff. 2024. Correcting misinformation on social media with a large language model.arXiv preprint arXiv:2403.11169(2024). Disclaimer: Note that some of the examples we showed may contain harmful or misleading content. 25 Zhang ...

    2024

  73. [81]

    It should not merely provide general context or an alternative viewpoint but must articulate how and why the post’s claim is inaccurate or misleading

    Direct and Specific Refutation: The comment must directly and explicitly address a specific claim made in the original post. It should not merely provide general context or an alternative viewpoint but must articulate how and why the post’s claim is inaccurate or misleading

  74. [82]

    good-faith attempt

    Verifiable Evidence or Source: The comment must provide direct, verifiable evidence or a credible source to support its refutation. This goes beyond a “good-faith attempt” to provide a basis. The evidence must be readily confirmable. • External Links: Links to reputable news o...

  75. [83]

    Not a Fact-check

    Synthesized Conclusion: The comment must provide a clear, concise summary of the corrective information. The comment itself should function as a mini-fact-check, presenting a conclusion that is easily understood by a lay audience without requiring them to visit the external li...

  76. [84]

    Identifies Inaccuracy: The note must explicitly or implicitly point out a factual inaccuracy, misleading claim, or lack of context in the original post

  77. [85]

    This includes, but is not limited to: •Offering Counter-Evidence: Providing specific facts, data, or links to sources that contradict the post

    Provides Substantiation: The note must substantiate its challenge with reasons, evidence, or relevant information. This includes, but is not limited to: •Offering Counter-Evidence: Providing specific facts, data, or links to sources that contradict the post. • Presenting Alter...

  78. [86]

    This is fake

    Mere Disagreement or Emotional Reaction: Notes that express simple agreement or disagreement (e.g., “This is fake”, “True”, “Propaganda”) without any accompanying reasoning or evidence

  79. [87]

    Irrelevant Information: Notes containing factual statements that are not relevant to the accuracy of the central claims in the post

  80. [88]

    Unsubstantiated Counter-Claims: Notes that make a different claim without providing any support

  81. [89]

    There was a thunderstorm in the area before the earthquake started. Plus, many electrical faults occur causing bright flashes. Not some crazy US attack

    Questions, Sarcasm, or General Discussion: Notes that do not contribute fact-checking information. 27 Zhang et al. B.3 Illustrative Examples The following examples demonstrate the application of these guidelines: Example 1: Useful Note Post: “The earthquake in Turkey looks lik...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.