Pith. sign in

REVIEW 3 major objections 4 minor 28 references

PANDA -- Paired Anti-hate Narratives Dataset from Asia: Using an LLM-as-a-Judge to Create the First Chinese Counterspeech Dataset

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces the first Mandarin Chinese counterspeech corpus, pairing each hate speech instance with generated and human-edited responses.

desk verdict A genuinely first Chinese counterspeech dataset attempt with an honest negative result about LLM-as-a-Judge bias, but the core pairing is only one-quarter human-verified and the verification rate is low, making the contribution a promising scaffold rather than a finished resource. read the letter →

arxiv 2501.00697 v2 pith:NE6YIATO submitted 2025-01-01 cs.CL

Michael Bennie , Demi Zhang , Bushi Xiao , Jing Cao , Chryseis Xinyi Liu , Jian Meng , Alayo Tripp This is my paper · ORCID
classification cs.CL
keywords counterspeechChinesehatespeechMandarinpaireddatasetLLM-as-a-judgesimulatedannealingannotationevaluationbias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to close a gap: counterspeech resources for Chinese are virtually nonexistent, and existing hate speech datasets are mostly English or Eurocentric. It introduces PANDA, a paired Mandarin hate-speech–counterspeech corpus of nearly 3,000 designed pairs, built from three open-source Chinese hate speech datasets. The construction pipeline uses an LLM-as-a-Judge to score and rank counterspeech candidates, a simulated annealing search over LLM-generated responses, and a round-robin tournament, followed by manual verification and editing. The paper also reports two findings that matter beyond the dataset itself: human annotators confirmed only 41.3% of the algorithmically selected hate speech entries as actual hate speech, and the LLM judge systematically ranked AI-generated paraphrases above human-edited counterspeech.

What carries the argument

The load-bearing mechanism is the scoring-and-search pipeline. Hate speech candidates are filtered by a Llama-3.1-70B hate score threshold (at least 51) and a string length threshold (at least 53 characters), then counterspeech candidates are generated by six LLMs and refined through simulated annealing, where acceptance probability follows a Boltzmann distribution over JudgeLM scores. JudgeLM is an LLM-based ranking method that scores relevance, fluency, and effectiveness; a round-robin tournament ranks the surviving candidates, and four per hate speech instance are passed to human annotators for selection and editing. The paper's evidence that JudgeLM ranks human-edited responses below AI paraphrases is what makes the pipeline's output depend on human oversight.

What would settle it

Randomly sample the final 2,974 pairs, have a diverse group of native Chinese annotators independently label each hate speech entry as hate speech, counterspeech, or neither, and compare with the pipeline's selection; if the confirmation rate does not rise above the 41.3% found in the first 785 instances, the filtering step has not isolated hate speech. Separately, on a held-out set of pairs, count how often JudgeLM ranks the human-preferred response first; the paper's t-tests predict it will rarely do so.

Watch

Extended reading notes

Core claim

The paper claims that it is possible to build the first Chinese counterspeech dataset by combining open-source hate speech corpora with an LLM-based scoring pipeline, and that doing so surfaces structural problems in both Chinese hate speech resources and LLM-based evaluation. Its central discovery is the paired PANDA corpus: hate speech instances selected from COLD, SWSR, and CHSD, each paired with four LLM-generated counterspeech candidates refined by human annotators. The accompanying analysis finds that existing Chinese hate speech labels are unreliable — a substantial share of labeled hate speech is actually counterspeech or neutral content — and that JudgeLM's rankings are misaligned with human preferences, favoring responses that restate or rephrase the original hate speech over responses that directly rebut its argument.

Load-bearing premise

The pipeline assumes that a Llama-3.1 hate score of at least 51 and a string length of at least 53 select genuine hate speech, but human annotators confirmed only 41.3% of the selected entries, so if that filtering assumption fails the dataset's paired structure is built on mislabeled inputs.

Editorial extensions

If this is right

  • The PANDA corpus gives Chinese-language hate speech and counterspeech research a paired resource where none existed, enabling direct study of counterspeech strategies in Mandarin.
  • If the reported judge bias is systematic, LLM-based evaluation of counterspeech in Chinese will need human-aligned scoring or multiple judges before it can be trusted.
  • The pipeline can be transferred to other East Asian languages with scarce counterspeech resources, such as Korean or Japanese, though cultural adaptation of annotation is required.
  • The low confirmation rate of hate speech labels in the source datasets argues for re-examining and re-annotating existing Chinese hate speech corpora before they are reused.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to use the 785 human-verified pairs as a held-out test set for calibrating LLM judges, measuring how often a judge's top pick matches the human-preferred response.
  • The 41.3% confirmation rate suggests that the amount of usable Chinese hate speech data may be far smaller than the raw corpus counts imply, so estimates of Chinese hate speech prevalence based on existing labels could be inflated.
  • If paraphrase-favoring bias generalizes, automated counterspeech systems evaluated only by LLM judges may drift toward echoing hate speech rather than rebutting it, which has consequences for how such systems are deployed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces PANDA, a dataset of paired Chinese hate speech and counterspeech (CS) entries, reportedly the first such resource for an East Asian language. The construction pipeline combines filtering from existing Chinese hate speech corpora using Llama-3.1 hate scores and string length thresholds, counterspeech generation via simulated annealing over multiple open-source LLMs, and a JudgeLM-based round-robin ranking step. Human annotators then scored the hate speech label of each entry and selected/edited the best CS response. Only 785 of the proposed 2,974 pairs received human annotation; of these, 41.3% were confirmed as hate speech, 31.0% as counterspeech, and 27.7% as neither. The paper also reports a statistical analysis suggesting that JudgeLM systematically downgrades human-preferred responses, interpreting this as a limitation of LLM-as-a-judge evaluation for Chinese counterspeech.

Significance. If the dataset were reliable, it would fill a genuine gap in non-English counterspeech resources and provide a valuable testbed for cross-lingual hate speech intervention research. The paper is also open about the limitations of existing Chinese hate speech labels and contributes a detailed description of the LLM-in-the-loop pipeline, which is useful for replication. Its explicit documentation of mislabeling rates and an LLM-based evaluation bias is a candid and useful lesson for the community. However, the central claim of a paired hate-speech/counterspeech corpus is currently not supported for the majority of the released pairs: only a small fraction has human verification, and even that verification reveals a low hate speech confirmation rate. The significance therefore hinges on whether the authors can either verify the remaining pairs or reframe the resource honestly. The JudgeLM bias analysis is interesting but methodologically confounded by the selection procedure, as noted below.

major comments (3)
  1. [§3.3, §3.6, Fig. 3] The central contribution—a paired hate-speech/counterspeech dataset—is undermined by the unverified hate speech labels. Section 3.3 selects 2,974 entries as hate speech using Llama-3.1 hate scores and a length threshold, but Figure 3 (reported in Section 3.6) shows that of the first 785 entries annotated by humans, only 41.3% were confirmed as hate speech, with 31.0% judged counterspeech and 27.7% neither. Since the remaining 2,189 pairs received no human verification, the released corpus likely contains a large fraction of non-hate-speech entries, contradicting the abstract's claim of a paired hate-speech/counterspeech resource. The paper must either verify all 2,974 pairs, release only the verified subset, or clearly reframe the resource as a noisy candidate set with confidence scores; the current abstract and contributions overstate what has been produced.
  2. [§3.5–§3.6] No inter-annotator agreement statistic is reported, despite four annotators independently scoring the same entries. Cohen's kappa or Krippendorff's alpha is standard for this type of annotation and is essential for assessing whether the 41.3% hate speech confirmation rate is a reliable estimate. Without it, even the human-verified subset cannot be used with confidence for training or evaluation, and the reported distribution in Figure 3 may be dominated by annotation idiosyncrasies.
  3. [§3.6, Table 3] The conclusion that JudgeLM is biased against human-preferred counterspeech is confounded by the selection procedure. The AI responses shown to annotators were themselves selected by JudgeLM through simulated annealing, so the comparison is not between independent human and AI candidates but between judge-selected candidates and human-edited versions of those same candidates. The one-sample t-test demonstrates a ranking difference but cannot support the attribution to 'goal misalignment' or 'bias' without controlling for this selection effect, for example by comparing against responses generated by a different judge or against random AI responses.
minor comments (4)
  1. [§4.1] Section 4.1 states that human annotators were able to create only 785 out of the proposed 2,974 pairs, but the released dataset is described elsewhere as containing 2,974 pairs. Please clarify what is actually released: 785 verified pairs, 2,974 unverified pairs, or a mixture with explicit flags.
  2. [Table 1] The corpus is abbreviated as 'CHSD' in the text but 'CSHD' in Table 1; please make the abbreviation consistent. Also, the 'Political' dataset (Wang et al., 2022) is listed in the table even though it is not open-source and not used in preprocessing; explain why it is included in the survey.
  3. [References] The reference for JudgeLM (Zubiaga et al., 2024) appears to have a corrupted author list ('Irune Zubiaga, Aitor Soroa, and Rodrigo Agerri' seems to be missing at least one author); please verify and correct this citation.
  4. [Abstract] The abstract contains a grammatical error: 'we introduce the a corpus' should be 'we introduce a corpus'. Please proofread the manuscript for similar typos.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pipeline is an empirical dataset-construction procedure whose key quality claims are checked against independent human annotation.

full rationale

The paper's construction chain is data assembly rather than a derivation that reduces to its own inputs. Hate-speech entries are selected from existing open-source datasets using source labels plus a Llama-3.1 score and length threshold (Section 3.3), and counterspeech is generated and ranked by an LLM-as-Judge (Section 3.4). No derived quantity is presented as a prediction from a fitted parameter: the threshold is an explicit corpus-construction choice ('we found that including strings that had a string length of at least 53 characters and a minimum hate score of 51 points provided a good balance'), and the paper's evaluative claims are tested against independent human annotation. Section 3.6 reports that annotators confirmed only 41.3% of 785 entries as hate speech, and Section 4.1 states that annotators 'were only able to create 785 out of the proposed 2,974 pairs of HS and CS,' which the paper treats as evidence of mislabeling in source corpora rather than as a validated output. The JudgeLM bias finding is supported by a one-sample t-test comparing model ranks with human-preferred answers (Table 3), so the model's ranking is not assumed correct by construction. No load-bearing argument rests on a self-citation from the present authors, and no uniqueness theorem or ansatz is imported from prior work of the same authors. The low 41.3% human-confirmation rate is a serious data-quality and validity limitation, but limitations and correctness risks are not circularity; the central claim of creating a paired Chinese HS-CS corpus is not forced by definition because the human annotation provides an external check on the model-assisted selection and ranking steps.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the filtering thresholds, which are fitted to data, and on the assumption that the LLM scores are meaningful. The paper provides human annotation as an external check, but it is only partial and reveals high noise.

free parameters (3)
  • Minimum hate score threshold = 51
    Chosen in Section 3.3 by maximizing log(AverageHateScore)*log(AverageTextLength)*NumInstances over possible subsets of the data.
  • Minimum string length threshold = 53
    Selected jointly with the hate score threshold in the same optimization in Section 3.3.
  • Simulated annealing hyperparameters = not reported
    The Boltzmann base B, number of iterations, and Hamming distance threshold d are introduced in Section 3.4.1 but their values are not given.
assumptions (4)
  • standard math The one-sample t-test assumptions are valid for comparing JudgeLM ranks to the baseline of 1.5.
    Used in Section 3.6 to claim systematic bias; assumes approximate normality and independence of observations, which is not explicitly checked.
  • domain assumption The functional definition of hate speech provided to annotators is a valid ground truth.
    Section 3.5 gives this definition as the basis for all human labels.
  • domain assumption The source datasets COLD, SWSR, and CHSD contain usable hate speech labels as an initial filter.
    Section 3.2 relies on these labels; the paper itself shows they are noisy, with only 41.3% confirmation by human annotators.
  • domain assumption JudgeLM scores reflect counterspeech quality well enough to guide selection.
    Section 3.4.1 uses JudgeLM to rank generated candidates; Section 3.6 shows it disagrees with human preferences, so this assumption is partially invalidated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PANDA -- Paired Anti-hate Narratives Dataset from Asia: Using an LLM-as-a-Judge to Create the First Chinese Counterspeech Dataset." pith.science (2026). https://pith.science/paper/NE6YIATO

@misc{pith2026250100697,
  author       = {Pith},
  title        = {Pith review of: PANDA -- Paired Anti-hate Narratives Dataset from Asia: Using an LLM-as-a-Judge to Create the First Chinese Counterspeech Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NE6YIATO}},
  note         = {Machine review of arXiv:2501.00697}
}
read the original abstract

Despite the global prevalence of Modern Standard Chinese language, counterspeech (CS) resources for Chinese remain virtually nonexistent. To address this gap in East Asian counterspeech research we introduce the a corpus of Modern Standard Mandarin counterspeech that focuses on combating hate speech in Mainland China. This paper proposes a novel approach of generating CS by using an LLM-as-a-Judge, simulated annealing, LLMs zero-shot CN generation and a round-robin algorithm. This is followed by manual verification for quality and contextual relevance. This paper details the methodology for creating effective counterspeech in Chinese and other non-Eurocentric languages, including unique cultural patterns of which groups are maligned and linguistic patterns in what kinds of discourse markers are programmatically marked as hate speech (HS). Analysis of the generated corpora, we provide strong evidence for the lack of open-source, properly labeled Chinese hate speech data and the limitations of using an LLM-as-Judge to score possible answers in Chinese. Moreover, the present corpus serves as the first East Asian language based CS corpus and provides an essential resource for future research on counterspeech generation and evaluation.

Figures

Figures reproduced from arXiv: 2501.00697 by the authors.

Figure 1
Figure 1. Proposed Data Processing Pipeline for Creat [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The scoring heat-map based on different com [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The distribution of human labeling on hate [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A histogram showing the ranking of human [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 9 canonical work pages

  1. [1]

    Catherine Buerger. 2021. https://doi.org/10.1177/20563051211063843 \#iamhere: Collective counterspeech and the quest to improve online discourse . Social Media + Society, 7(4):20563051211063843

    work page doi:10.1177/20563051211063843 2021

  2. [2]

    Bianca Cepollaro, Maxime Lepoutre, and Robert Mark Simpson. 2023. https://doi.org/10.1111/phc3.12890 Counterspeech . Philosophy Compass, 18(1):e12890

    doi:10.1111/phc3.12890 2023

  3. [3]

    Yi-Ling Chung, Gavin Abercrombie, Florence Enock, Jonathan Bright, and Verena Rieser. 2023. https://arxiv.org/abs/2307.04761 Understanding counterspeech for online harm mitigation . Preprint, arXiv:2307.04761

    arXiv 2023

  4. [5]

    Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019 b . https://doi.org/10.18653/v1/P19-1271 CONAN - CO unter NA rratives through nichesourcing: a multilingual dataset of responses to fight online hate speech . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2819--2829, Floren...

    doi:10.18653/v1/p19-1271 2019

  5. [6]

    Ona de Gibert, Naiara Perez, Aitor Garc \' a-Pablos, and Montse Cuadros. 2018. https://doi.org/10.18653/v1/W18-5102 Hate speech dataset from a white supremacy forum . In Proceedings of the 2nd Workshop on Abusive Language Online ( ALW 2) , pages 11--20, Brussels, Belgium. Association for Computational Linguistics

    doi:10.18653/v1/w18-5102 2018

  6. [7]

    Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, and Minlie Huang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.796 COLD : A benchmark for C hinese offensive language detection . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11580--11599, Abu Dhabi, United Arab Emirates. Associat...

    doi:10.18653/v1/2022.emnlp-main.796 2022

  7. [8]

    Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiro g lu, and Marco Guerini. 2021. https://doi.org/10.18653/v1/2021.acl-long.250 Human-in-the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa...

    doi:10.18653/v1/2021.acl-long.250 2021

  8. [9]

    Aiqi Jiang, Xiaohan Yang, Yang Liu, and Arkaitz Zubiaga. 2022. Swsr: A chinese dataset and lexicon for online sexism detection. Online Social Networks and Media, 27:100182

    work page 2022

Show all 28 references
  1. [10]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  2. [11]

    Junyu Lu, Bo Xu, Xiaokun Zhang, Changrong Min, Liang Yang, and Hongfei Lin. 2023. https://doi.org/10.18653/v1/2023.acl-long.898 Facilitating fine-grained detection of C hinese toxic language: Hierarchical taxonomy, resources, and benchmarks . In Proceedings of the 61st Annual ...

    2023 doi

  3. [12]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  4. [13]

    Aashima Poudhar, Ioannis Konstas, and Gavin Abercrombie. 2024. https://doi.org/10.18653/v1/2024.woah-1.20 A strategy labelled dataset of counterspeech . In Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), pages 256--265, Mexico City, Mexico. Association f...

    2024 doi

  5. [14]

    Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. https://doi.org/10.18653/v1/D19-1482 A benchmark dataset for learning to intervene in online hate speech . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process...

    2019 doi

  6. [15]

    Anish Kumar Raj Ratn Pranesh, Ambesh Shekhar. 2020. Towards automatic online hate speech intervention generation using pretrained language model. Anonymous preprint under review

    2020

  7. [16]

    Xiaojun Rao, Yangsen Zhang, Qilong Jia, Xueyang Liu, Xiaojun Rao, Yangsen Zhang, Shuang Peng, Qilong Jia, and Xueyang Liu. 2023. https://aclanthology.org/2023.ccl-1.44 Research on chinese hate speech detection method based on R o BERT a ( C hinese hate speech detection method ...

    2023

  8. [17]

    Punyajoy Saha, Aalok Agrawal, Abhik Jana, Chris Biemann, and Animesh Mukherjee. 2024. https://aclanthology.org/2024.lrec-main.1090 On zero-shot counterspeech generation by LLM s . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language ...

    2024

  9. [18]

    Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2021. On the safety of conversational models: Taxonomy, dataset, and benchmark. arXiv preprint arXiv:2110.08466

    2021 arXiv

  10. [19]

    Serra Sinem Tekiro g lu, Helena Bonaldi, Margherita Fanton, and Marco Guerini. 2022. https://doi.org/10.18653/v1/2022.findings-acl.245 Using pre-trained language models for producing counter narratives against hate speech: a comparative study . In Findings of the Association f...

    2022 doi

  11. [20]

    Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. https://doi.org/10.18653/v1/W19-3509 Challenges and frontiers in abusive content detection . In Proceedings of the Third Workshop on Abusive Language Online, pages 80--93, Florence,...

    2019 doi

  12. [21]

    Chih-Chien Wang, Min-Yuh Day, and Chun-Lian Wu. 2022. https://doi.org/10.1109/ACCESS.2022.3160712 Political hate speech detection and lexicon building: A study in taiwan . IEEE Access, 10:44337--44346

    2022

  13. [22]

    Ke Wang and Xiaojun Wan. 2018. https://api.semanticscholar.org/CorpusID:51609768 Sentigan: Generating sentimental texts via mixture adversarial networks . In International Joint Conference on Artificial Intelligence

    2018

  14. [23]

    Tao Zhang, Ziqian Zeng, Yuxiang Xiao, Huiping Zhuang, Cen Chen, James Foulds, and Shimei Pan. 2024. https://arxiv.org/abs/2406.13925 Genderalign: An alignment dataset for mitigating gender bias in large language models . Preprint, arXiv:2406.13925

    2024 arXiv

  15. [24]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with BERT . CoRR, abs/1904.09675

    2019 arXiv

  16. [25]

    Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.262 Towards identifying social bias in dialog systems: Framework, dataset, and benchmark . In Findings of the Ass...

    2022 doi

  17. [26]

    Wanzheng Zhu and Suma Bhat. 2021. https://arxiv.org/abs/2106.01625 Generate, prune, select: A pipeline for counterspeech generation against online hate speech . CoRR, abs/2106.01625

    2021 arXiv

  18. [27]

    Irune Zubiaga, Aitor Soroa, and Rodrigo Agerri. 2024. https://arxiv.org/abs/2406.15227 A llm-based ranking method for the evaluation of automatic counter-narrative generation . Preprint, arXiv:2406.15227

    2024 arXiv

  19. [28]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  20. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.