The reviewed record of science sign in
Pith

arxiv: 2410.07053 · v1 · pith:YOBWPMKF · submitted 2024-10-09 · cs.HC · cs.CL

Robots in the Middle: Evaluating LLMs in Dispute Resolution

Jinzhe Tan , Hannes Westermann , Nikhil Reddy Pottanigari , Jarom\'ir \v{S}avelka , S\'ebastien Mee\`us , Mia Godet , Karim Benyekhlef This is my paper

Reviewed by Pithpith:YOBWPMKFopen to challenge →

classification cs.HC cs.CL
keywords llmsdisputeinterventionmessagesresolutionableacrossannotators
0
0 comments X
read the original abstract

Mediation is a dispute resolution method featuring a neutral third-party (mediator) who intervenes to help the individuals resolve their dispute. In this paper, we investigate to which extent large language models (LLMs) are able to act as mediators. We investigate whether LLMs are able to analyze dispute conversations, select suitable intervention types, and generate appropriate intervention messages. Using a novel, manually created dataset of 50 dispute scenarios, we conduct a blind evaluation comparing LLMs with human annotators across several key metrics. Overall, the LLMs showed strong performance, even outperforming our human annotators across dimensions. Specifically, in 62% of the cases, the LLMs chose intervention types that were rated as better than or equivalent to those chosen by humans. Moreover, in 84% of the cases, the intervention messages generated by the LLMs were rated as better than or equal to the intervention messages written by humans. LLMs likewise performed favourably on metrics such as impartiality, understanding and contextualization. Our results demonstrate the potential of integrating AI in online dispute resolution (ODR) platforms.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations

    cs.AI 2026-06 unverdicted novelty 6.0

    SoCRATES introduces a benchmark for proactive LLM mediators across eight domains and five socio-cognitive axes with topic-localized evaluation, finding top models close only about one-third of the unmediated consensus gap.

  2. Real-Time Group Dynamics with LLM Facilitation: Evidence from a Charity Allocation Task

    cs.HC 2026-05 unverdicted novelty 6.0

    LLM facilitators in real-stakes group charity decisions shift specific allocations without raising consensus or participation equity, yet increase perceived trust and preference for the process.

  3. Can You Trust What You See? Human and AI Detection of Synthetic Legal Evidence

    cs.CV 2026-05 unverdicted novelty 5.0

    Humans reach 64.8% accuracy detecting synthetic legal evidence images overall but drop to chance levels on top generators, while MLLMs achieve 100% specificity yet only 5.9% detection on the hardest synthetics, with u...

  4. Real-Time Group Dynamics with LLM Facilitation: Evidence from a Charity Allocation Task

    cs.HC 2026-05 unverdicted novelty 5.0

    LLM facilitation in group charity allocation leaves consensus and participation equity unchanged while shifting specific allocations up to 5.5 points and increasing perceived trust.