Pith. sign in

REVIEW 3 major objections 4 minor 19 references

Leveraging a Multi-Agent LLM-Based System to Educate Teachers in Hate Incidents Management

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multi-agent LLM system with retrieval and modelled student personas detects explicit hate speech at 79% accuracy and, in a pilot with teachers, was reported to deepen their understanding of context-dependent hate cases.

desk verdict A well-specified multi-agent LLM system for hate-incident teacher training, with honest but thin benchmarks; the educational efficacy claim is not yet measured. read the letter →

arxiv 2506.23774 v1 pith:TMFA4PBE submitted 2025-06-30 cs.CY cs.HC

Ewelina Gajewska , Michal Wawer , Katarzyna Budzynska , Jaros{l}aw A. Chudziak This is my paper · ORCID
classification cs.CYcs.HC
keywords multi-agentLLMsystemteachereducationhatespeechdetectionretrieval-augmentedgenerationpersonamodellingimplicitincidentmanagement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a multi-agent large language model system, built from retrieval-augmented prompting and modelled student personas, can serve as a practice ground for teachers learning to handle hate incidents in school. It reports that the system detects explicit hate speech with up to 79% accuracy on a standard benchmark and implicit hate speech with up to 60%, with the multi-agent retrieval setup besting single-agent and non-retrieval versions. In a pilot workshop with professional teachers, participants reported that the system helped them see how annotator disagreements and context shape hate-speech interpretation. The central educational claim is that such systems can give teachers safe, scalable, and personalisable training that traditional seminars do not easily provide.

What carries the argument

ARISE, the system at the heart of the paper, combines a manager agent modelled as an academic professor with three student agents modelled with distinct backstories in psychology, pedagogy, and cognitive science, plus cultural-lens advisory agents and a retrieval-augmented generation module that supplies case studies and legal definitions. The mechanism is that each persona biases the underlying LLM's reading of an incident, and the manager agent aggregates and reconciles those readings, while retrieval adds external context that sharpens classification. That multi-perspective aggregation is what carries the argument that the system captures the subjectivity and context-dependence of hate speech.

What would settle it

Run a controlled study in which one group of teachers practises on ARISE with classroom vignettes and a similar group receives only written materials, with blinded pre/post scoring of their proposed interventions; if the ARISE group shows no greater improvement in identifying implicit hate speech than the control group, the educational claim is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that an agentic LLM platform combining retrieval-augmented generation with persona-modelled agents can both classify hate speech and teach teachers about its interpretation. On 100-sample evaluations, the system reaches 79% accuracy on explicit hate speech (HateXplain) and 60% on implicit hate speech (Latent Hatred), with the multi-agent RAG configuration outperforming single agents and non-RAG baselines. In a workshop case study with professional teachers, participants reported a deeper understanding of the context-dependence of hate speech and of why annotators disagree, and rated the system's analysis satisfactory while noting occasional over-interpretation and a need for legal-norm comparisons. The intended contribution is a demonstrator of LLM-based, low-cost, continuous teacher training on hate incidents.

Load-bearing premise

The educational claim that teachers learn from the system rests on self-reported interview responses from an unspecified number of teachers, with no pre/post measures or control group (Section 4); if those reports do not reflect real skill gains, the central classroom-training claim collapses.

Editorial extensions

If this is right

  • If the results hold, teacher-training programmes can supplement seminars with a low-cost, scalable simulation in which teachers practise on diverse hate-incident scenarios at any time.
  • The consistent multi-agent advantage suggests that forcing several modelled perspectives on the same incident yields better hate-speech classification than a single model, especially on implicit cases.
  • Using the system in workshops can make annotator disagreement a teachable topic: teachers see that the same words can be interpreted differently depending on context and identity.
  • The accuracy gap between explicit (79%) and implicit (60%) cases marks a practical boundary: educators can rely on the tool for clear-cut incidents but should treat ambiguous ones as flags for human judgment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated consequence of the accuracy comparison is that the multi-agent advantage may extend beyond hate speech to any classification task where context and annotator disagreement matter; the paper only demonstrates it on two hate-speech benchmarks.
  • A natural follow-up the paper does not run is a controlled pre/post workshop study that measures actual intervention choices rather than reported understanding.
  • The teacher request for a legal-norm comparison module suggests a product path: extending ARISE with jurisdiction-specific hate-speech rules would turn it from an interpretation aid into a decision-support tool, but that requires legal validation the authors defer.
  • Because the system models student personas, the same architecture could plausibly be used with students themselves as learners, practising recognition of context before an incident occurs; the paper's workshop targets only teachers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ARISE, a multi-agent LLM-based system that combines retrieval-augmented generation (RAG) and persona modeling to help teachers practice managing hate incidents in schools. The authors report classification accuracy of the underlying GPT-o1-mini system on 100-sample subsets of HateXplain and Latent Hatred across single- and multi-agent configurations, with and without RAG, and describe a pilot workshop in which teachers interacted with the system and were interviewed afterwards. The central claim is that this pilot demonstrates teachers' enhanced understanding of hate speech interpretation, annotator disagreements, and intervention strategies.

Significance. If the educational claim were properly supported, the paper would make a useful contribution by demonstrating a concrete application of multi-agent LLM systems to a socially important training domain, and by providing initial evidence about the effect of persona modeling and RAG on hate speech classification. The comparison of single-agent versus multi-agent setups and the use of two benchmarks with different difficulty levels are valuable design choices, and the qualitative findings suggest relevant requirements such as the need for legal-norm contextualization. However, the current evidence base is too thin: the benchmark results lack uncertainty quantification, and the pilot rests entirely on post-hoc self-reports with no measured learning outcome.

major comments (3)
  1. [Section 4; Abstract; Section 5] The headline claim that 'pilot evaluation demonstrates teachers' enhanced understanding' is not supported by the reported evidence. The evaluation consists exclusively of semi-structured interviews conducted after the workshop, and the number of participating teachers, the interview protocol, and the method of qualitative analysis are all unspecified. There is no pre/post measure, no control condition, no objective outcome such as blinded expert scoring of intervention plans, and no coding scheme or inter-rater reliability. The reported outcomes are compatible with social desirability and general satisfaction rather than with measurable learning. Because the stated purpose of the system is teacher education, this unmeasured learning gain is load-bearing; the claim should be substantially softened or replaced with a properly designed evaluation.
  2. [Section 3.3, Table 1] The classification results are based on 100 samples per dataset with no confidence intervals, error bars, repeated runs, or significance tests. Claims such as 'the multi-agent setup consistently outperforms single-agent configurations' and RAG improvements of 4.5 p.p. or 1-7 p.p. are therefore statistically unsupported; differences of a few percentage points on n=100 could easily arise from sampling noise. In addition, RQ1 asks how the machine compares to humans in annotating hate speech, but no human annotation comparison is reported anywhere. The paper should report variance across samples or runs, significance testing or confidence intervals, and either a human-comparison baseline or an explicit revision of RQ1.
  3. [Section 4 (RQ4)] RQ4 asks for criteria for evaluating the usefulness and effectiveness of the platform in an educational context, but no such criteria are defined or operationalized. The evaluation reports that teachers rated the system's analysis as satisfactory and noted strengths and weaknesses, but it never states what would count as usefulness or effectiveness, how these constructs were measured, or how the interview data were analyzed (e.g., thematic analysis with systematic coding). Without these criteria and a transparent analysis procedure, the conclusions about the platform's educational value cannot be assessed.
minor comments (4)
  1. [Section 3.3, Table 1] The table header contains 'T able 1' and mixes capitalization styles such as 'gpt-o1-mini'; the caption also lacks the sample size and the number of runs, which should be stated explicitly.
  2. [Section 3.1] The text refers to Figure 1 as showing the ARISE architecture, but no figure appears in the manuscript text; either include the figure or remove the reference.
  3. [References] References [4], [6], and [9] are self-citations to prior multi-agent work; the text should make explicit how each prior contribution is used in the ARISE architecture rather than listing them without connection.
  4. [Throughout] There are typographical errors, including 'humouristically' (Section 4), 'S,tefănit,ă' in reference [19], and inconsistent notation 'RQ.1' vs. 'RQ1'; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark evaluation is external and the teacher pilot, while methodologically weak, does not reduce by construction to the system's own outputs.

full rationale

The paper's classification claims are evaluated against two external benchmark datasets, HateXplain and Latent Hatred, with no fitted parameters and no quantity derived from the model's own outputs; the reported accuracies are externally grounded comparisons. The teacher-education claim rests on post-workshop semi-structured interviews, but the paper does not define 'enhanced understanding' in terms of the system's internal outputs, so the conclusion is not equivalent to its input by construction; it is an evidentiary limitation rather than a circular derivation. The self-citations (refs. 4, 6, 9, all involving co-author Chudziak) support background claims about multi-agent LLM systems and prompt engineering, but those claims are not load-bearing for the paper's central empirical results, which are independently benchmarked. No equation, definition, or fitted parameter is reused as a prediction, and no uniqueness claim is imported from the authors' prior work. Accordingly, no specific circular step can be quoted.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The central claims rely on the assumed quality of LLM outputs and the transferability of benchmark results and self-reports to real-world teacher competence; these assumptions are not independently tested.

assumptions (3)
  • domain assumption GPT-o1-mini is capable of producing realistic and pedagogically useful simulations of hate incidents.
    The entire system depends on the LLM's ability to role-play personas and analyze hate speech; this is asserted in Section 3.2 but not independently validated.
  • domain assumption Classification accuracy on HateXplain and Latent Hatred benchmarks is representative of real-world performance in teacher training contexts.
    The paper uses benchmark accuracy as evidence of the system's reliability for education, but does not show transfer to classroom settings (Section 3.3).
  • domain assumption Self-reported interview data from a small pilot accurately reflects actual learning and skill development.
    The conclusion of enhanced understanding in Section 4 is based on qualitative self-reports without pre/post testing or objective measures.
invented entities (1)
  • ARISE platform (multi-agent LLM system)
    purpose: Simulates hate incidents and provides analyses for teacher training
    The system's efficacy is only supported by the paper's own pilot evaluation; no external validation or falsifiable prediction outside this work is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging a Multi-Agent LLM-Based System to Educate Teachers in Hate Incidents Management." pith.science (2026). https://pith.science/paper/TMFA4PBE

@misc{pith2026250623774,
  author       = {Pith},
  title        = {Pith review of: Leveraging a Multi-Agent LLM-Based System to Educate Teachers in Hate Incidents Management},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TMFA4PBE}},
  note         = {Machine review of arXiv:2506.23774}
}
read the original abstract

Computer-aided teacher training is a state-of-the-art method designed to enhance teachers' professional skills effectively while minimising concerns related to costs, time constraints, and geographical limitations. We investigate the potential of large language models (LLMs) in teacher education, using a case of teaching hate incidents management in schools. To this end, we create a multi-agent LLM-based system that mimics realistic situations of hate, using a combination of retrieval-augmented prompting and persona modelling. It is designed to identify and analyse hate speech patterns, predict potential escalation, and propose effective intervention strategies. By integrating persona modelling with agentic LLMs, we create contextually diverse simulations of hate incidents, mimicking real-life situations. The system allows teachers to analyse and understand the dynamics of hate incidents in a safe and controlled environment, providing valuable insights and practical knowledge to manage such situations confidently in real life. Our pilot evaluation demonstrates teachers' enhanced understanding of the nature of annotator disagreements and the role of context in hate speech interpretation, leading to the development of more informed and effective strategies for addressing hate in classroom settings.

Figures

Figures reproduced from arXiv: 2506.23774 by the authors.

Figure 1
Figure 1. Architecture of the ARISE platform [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 12 canonical work pages

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    International Journal of Modern Education and Computer Science16(4), 71–86 (2024)

    Alshahrani, K., Qureshi, R.J.: Review the prospects and obstacles of ai-enhanced learning environments: The role of chatgpt in education. International Journal of Modern Education and Computer Science16(4), 71–86 (2024)

    work page 2024

  3. [3]

    Social Psy- chology of Education pp

    Bilz, L., Fischer, S.M., Kansok-Dusche, J., Wachs, S., Wettstein, A.: Teachers’ intervention strategies for handling hate-speech incidents in schools. Social Psy- chology of Education pp. 1–24 (2024)

    work page 2024

  4. [4]

    In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering

    Cinkusz, K., Chudziak, J.A.: Towards llm-augmented multiagent systems for ag- ile software engineering. In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. p. 2476–2477. Association for Computing Machinery (2024), https://doi.org/10.1145/3691620.3695336

    arXiv 2024

  5. [5]

    In: Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing

    ElSherief, M., Ziems, C., Muchlinski, D., Anupindi, V., Seybolt, J., De Choud- hury, M., Yang, D.: Latent hatred: A benchmark for understanding implicit hate speech. In: Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing. pp. 345–363. Association for Computa- tional Linguistics, Online and Punta Cana, Dominican Republ...

    work page 2021

  6. [6]

    In: Proceedings of the ACIIDS 2025 : 17th Asian Conference on Intelligent Information and Database Systems (2025)

    Harbar, Y., Chudziak, J.A.: Simulating oxford-style debates with llm-based multi- agent systems. In: Proceedings of the ACIIDS 2025 : 17th Asian Conference on Intelligent Information and Database Systems (2025)

    work page 2025

  7. [7]

    Trauma, violence, & abuse 24(4), 2598–2615 (2023)

    Kansok-Dusche,J.,Ballaschk,C.,Krause,N.,Zeißig,A.,Seemann-Herz,L.,Wachs, S., Bilz, L.: A systematic review on hate speech among children and adolescents: Definitions, prevalence, and overlap with related phenomena. Trauma, violence, & abuse 24(4), 2598–2615 (2023)

    work page 2023

  8. [8]

    Education and Information Technologies29(7), 8693–8724 (2024)

    Kim, J.: Leading teachers’ perspective on teacher-ai collaboration in education. Education and Information Technologies29(7), 8693–8724 (2024)

    work page 2024

Show all 19 references
  1. [9]

    In: Proceedings of the 38th Pacific AsiaConferenceonLanguage,InformationandComputation.AssociationforCom- putational Linguistics (Dec 2024)

    Kostka, A., Chudziak, J.A.: Synergizing logical reasoning, knowledge management and collaboration in multi-agent llm system. In: Proceedings of the 38th Pacific AsiaConferenceonLanguage,InformationandComputation.AssociationforCom- putational Linguistics (Dec 2024)

    2024

  2. [10]

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive nlp tasks (2021), https://arxiv.org/abs/2005.11401

    2021 arXiv

  3. [11]

    In: Proceedings of the AAAI conference on artificial intelligence

    Mathew,B.,Saha,P.,Yimam,S.M.,Biemann,C.,Goyal,P.,Mukherjee,A.:Hatex- plain: A benchmark dataset for explainable hate speech detection. In: Proceedings of the AAAI conference on artificial intelligence. vol. 35, pp. 14867–14875 (2021)

    2021

  4. [12]

    Gajewska et al

    Mondal, H., Marndi, G., Behera, J.K., Mondal, S.: Chatgpt for teachers: Practical examplesforutilizingartificialintelligenceforeducationalpurposes.IndianJournal of Vascular and Endovascular Surgery (2023) 8 E. Gajewska et al

    2023

  5. [13]

    Frontiers in Virtual Reality2, 672003 (2021)

    Oyekoya, O., Urbanski, J., Shynkar, Y., Baksh, A., Etsaghara, M.: Exploring first- person perspectives in designing a role-playing vr simulation for bullying preven- tion: A focus group study. Frontiers in Virtual Reality2, 672003 (2021)

    2021

  6. [14]

    TechTrends pp

    Pitura, J., Kaplan-Rakowski, R., Asotska-Wierzba, Y.: The vr-ai–assisted sim- ulation for content knowledge application in pre-service efl teacher training. TechTrends pp. 1–11 (2024)

    2024

  7. [15]

    IEEE Access 12, 26839–26874 (2024)

    Raiaan, M.A.K., Mukta, M.S.H., Fatema, K., Fahad, N.M., Sakib, S., Mim, M.M.J., Ahmad, J., Ali, M.E., Azam, S.: A review on large language models: Architectures, applications, taxonomies, open issues and challenges. IEEE Access 12, 26839–26874 (2024). https://doi.org/10.1109/A...

    2024

  8. [16]

    In: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication/Social Media

    Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., Wojatzki, M.: Mea- suring the reliability of hate speech annotations: The case of the european refugee crisis. In: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication/Social Media. pp. 6–...

    2016

  9. [17]

    In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V

    Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., Smith, N.A.: Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In: Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V. (eds.) Proceedings of the 2022 Conference of the North Amer...

    2022 doi

  10. [18]

    arXiv:2309.14365v1 [cs.CL] (9 2023)

    Zhao, P., Jin, Z., Cheng, N.: An in-depth survey of large language model-based artificial intelligence agents. arXiv:2309.14365v1 [cs.CL] (9 2023)

    2023 arXiv

  11. [19]

    Romanian Journal of Communication and Public Relations23(1), 47–55 (2021)

    S,tefănit,ă, O., Buf, D.M.: Hate speech in social media and its effects on the LGBT community: A review of the current research. Romanian Journal of Communication and Public Relations23(1), 47–55 (2021)

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.