Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
read the original abstract
Recent advancements in large language models (LLMs) underscore their potential for responding to inquiries in various domains. However, ensuring that generative agents provide accurate and reliable answers remains an ongoing challenge. In this context, multi-agent debate (MAD) has emerged as a promising strategy for enhancing the truthfulness of LLMs. We benchmark a range of debating and prompting strategies to explore the trade-offs between cost, time, and accuracy. Importantly, we find that multi-agent debating systems, in their current form, do not reliably outperform other proposed prompting strategies, such as self-consistency and ensembling using multiple reasoning paths. However, when performing hyperparameter tuning, several MAD systems, such as Multi-Persona, perform better. This suggests that MAD protocols might not be inherently worse than other approaches, but that they are more sensitive to different hyperparameter settings and difficult to optimize. We build on these results to offer insights into improving debating strategies, such as adjusting agent agreement levels, which can significantly enhance performance and even surpass all other non-debate protocols we evaluated. We provide an open-source repository to the community with several state-of-the-art protocols together with evaluation scripts to benchmark across popular research datasets.
This paper has not been read by Pith yet.
Forward citations
Cited by 8 Pith papers
-
MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
MARGIN is an online per-agent per-band calibration method using symmetric exponentially weighted moving averages with Bayesian shrinkage that reduces calibration error 3-6x under distribution shift and improves multi-...
-
Collective Alignment in LLM Multi-Agent Systems: Disentangling Bias from Cooperation via Statistical Physics
LLM multi-agent systems on lattices show bias-driven order-disorder crossovers instead of true phase transitions, with extracted effective couplings and fields serving as model-specific fingerprints.
-
EquiMem: Calibrating Shared Memory in Multi-Agent Debate via Game-Theoretic Equilibrium
EquiMem calibrates shared memory in multi-agent debate by computing a game-theoretic equilibrium from agent queries and paths, outperforming heuristics and LLM validators across benchmarks while remaining robust to ad...
-
Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debate
SDRL trains LLMs via self-generated multi-path debates and joint optimization of standalone plus debate-conditioned responses to boost both single-model reasoning and multi-agent debate performance.
-
To Isolate or to Score? Model-Adaptive Assessment for Cost-Efficient Multi-Agent RAG
Empirical study finds isolation drives gains for weak models in multi-agent RAG while scoring matters for strong ones, enabling MADARA for cost-efficient adaptive assessment.
-
MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
MARGIN is an online calibration technique using symmetric EWMA and Bayesian shrinkage that learns per-agent per-band factors from the task stream, cutting calibration error 3-6x versus design-time baselines and liftin...
-
Representational Collapse in Multi-Agent LLM Committees: Measurement and Diversity-Aware Consensus
LLM agent committees exhibit representational collapse with mean cosine similarity of 0.888, and diversity-aware consensus reaches 87% accuracy on GSM8K versus 84% for self-consistency at lower cost.
-
BLUEmed: Retrieval-Augmented Multi-Agent Debate for Clinical Error Detection
BLUEmed combines hybrid RAG with structured multi-agent debate and a safety filter to detect terminology substitution errors in clinical notes, reaching 69.13% accuracy under few-shot prompting and outperforming singl...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.