REVIEW 3 major objections 4 minor 1 cited by
Interaction Protocol Shapes Moral Judgment in Multi-Agent Debate
T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read This paper shows that whether language models revise their moral verdicts in multi-agent debate depends less on which model they are than on how the debate is orchestrated: parallel ('synchronous') exchanges make GPT-4.1 nearly immovable, w
desk verdict The headline effect is real and worth taking seriously, but the inertia/conformity decomposition is less secure than it looks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a multinomial logistic model: for each model m, dilemma d, and round r, the log-odds of a verdict depend on that model's baseline preferences, a fixed dilemma effect, an inertia term α_m for repeating its own previous verdict, and two conformity terms γ_prev,m (frequency of a verdict in earlier rounds) and γ_within,m (frequency within the current round, non-zero only in round-robin). The two deliberation protocols—synchronous (parallel, simultaneous responses) and round-robin (sequential, with later models seeing earlier verdicts)—are the manipulated variable that makes inertia and conformity separable. Values are classified via a curated 48-value taxonomy and measur
What would settle it
Re-run the 1,000-dilemma experiment, say 10 times per dilemma at temperature 1, and compute change-of-verdict rates and the fitted α and γ parameters with bootstrapped confidence intervals. If GPT-4.1's synchronous change-of-verdict rate overlaps Claude's, or if its round-robin within-round conformity odds ratio falls below Gemini's, the paper's central claim that protocol flips model flexibility would be falsified.
Extended reading notes
Core claim
The authors claim that verdict revision and consensus in LLM deliberation are governed by two opposing forces—inertia (repeating one's own prior verdict) and conformity (yielding to verdicts seen from others)—and that the balance of these forces is format-dependent. In synchronous debate, GPT-4.1's change-of-verdict rate is 0.6–3.1% while Claude and Gemini revise 28–41% of the time; in round-robin debate, GPT-4.1 and Gemini conform strongly to the verdict they see first, with GPT-4.1's within-round conformity odds ratio at 8.68. Consensus is more likely when models' invoked values converge, and value similarity (Jaccard index over a 48-value taxonomy) rises by 30–60% in deliberations that re
Load-bearing premise
Every headline number rests on a single run per dilemma at a temperature of 1; because language-model output is stochastic, a repeat run could shift the revision rates and the fitted inertia/conformity parameters, possibly changing the relative ordering of models.
Editorial extensions
If this is right
- Consensus in multi-agent moral deliberation often reflects conformity or inertia rather than genuine persuasion, so 'consensus' is not evidence of correctness.
- The same model can flip from rigid to compliant merely by switching the interaction protocol, so system designers should test values under both formats before deployment.
- Value convergence closely tracks verdict convergence: when models agree they share roughly three of five values, and consensus-reaching deliberations raise Jaccard similarity by 30–60%.
- System prompt steering ('balance consensus and correctness') increases verdict changes but does not raise consensus rates, meaning models often move to different verdicts rather than converging.
- Order effects in round-robin deliberation are large enough to shift the final verdict distribution—for example, GPT steering over 70% of dilemmas to NTA when going first in a three-way debate.
Reading between the lines
- An implication the paper leaves implicit: a moral-advice system that uses synchronous parallel debates may overstate a model's confidence, while a round-robin system may silently inherit the first model's bias—so a hybrid protocol could serve as a calibrating middle ground.
- Because the paper runs each dilemma once at temperature 1, the headline odds ratios are point estimates; a natural following experiment would repeat the same 1,000 dilemmas across many seeds to measure variance and confirm the model-ordering results.
- The value-inheritance result (Claude and Gemini often inherit GPT's personal-autonomy values, while GPT rarely inherits empathy values) hints that influence flows asymmetrically—worth testing whether that asymmetry persists if models exchange system prompts or are prompted to match the other's style.
- An untested extension: head-to-head and three-way formats produced different consensus rates; larger panels (five or more agents) or mixed human-AI panels could reveal whether the format effect saturates or reverses.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares two multi-agent deliberation protocols—synchronous and round-robin—for three proprietary LLMs (GPT-4.1, Claude 3.7 Sonnet, Gemini 2.0 Flash) across 1,000 everyday moral dilemmas from r/AmItheAsshole. It reports change-of-verdict (CoV) rates, consensus rates, value-usage and value-inheritance patterns, and fits a multinomial logistic model to separate 'inertia' and 'conformity' parameters. The central finding is that deliberation format changes LLMs' willingness to revise moral verdicts and that this is model-dependent: GPT shows high inertia in synchronous settings but high conformity in round-robin settings, while Claude and Gemini behave differently. The paper also tests a modified system prompt aimed at steering consensus-seeking behavior.
Significance. If the findings hold, they establish that interaction protocol is a first-order variable in multi-agent moral judgment rather than a secondary implementation detail, with implications for deployed multi-agent systems. The study uses a large, naturalistic dilemma corpus, releases code, and reports bootstrapped confidence intervals. The aggregate CoV differences (e.g., 0.6% vs. 41.2%) are large and internally consistent. However, the manuscript's central qualitative claim is more robust than its derived quantitative decomposition: the raw CoV rates have tight binomial confidence intervals despite single-run data, whereas the fitted inertia/conformity parameters and the value-level analyses rest on single-sample, unvalidated measurements.
major comments (3)
- [§5, §4.1, Table 1, Eq. (2)] The single-run design is acknowledged in §5 ('we ran each experiment once'). This is not fatal for the headline CoV rates: for N=1,000, the binomial 95% CI for 0.6% is roughly [0.1%, 1.1%] and for 41.2% roughly [38.2%, 44.2%], so the aggregate ordering is robust. However, the multinomial model in Eq. (2) estimates α and γ parameters from those same single runs and treats each round as independent. Verdict flips at temperature 1 are serially correlated within a deliberation, and the ~4,000 dilemma fixed effects φ_dv are not accompanied by convergence, identifiability, or model-check diagnostics. Consequently, Table 1's odds ratios (α_GPT 8.27, γ_within,GPT 8.68) may overstate precision and the inertia/conformity split could absorb the same 'conform-then-stick' behavior. Please provide repeated runs (or a variance decomposition), cluster-robust standard errors, and posterior predictive or
- [§3.3, Figs. 4–7] The value classification uses a single LLM judge (Gemini 2.5 Flash) with a single pass per response, no inter-rater reliability, and no repeated sampling. This is load-bearing for the claims that certain values are 'inherited' and that value alignment drives consensus. Since the judge is itself a Gemini-family model and the deliberators include Gemini 2.0 Flash, systematic judge bias could affect the value-occurrence and value-similarity results. The reported bootstrapped CIs only capture variation over dilemmas, not variation over judge draws or judge identity. Please add multi-judge or repeated-judge measurements, agreement statistics (e.g., Cohen's κ or a judge-replication analysis), and show that the value-level conclusions are stable under judge variation.
- [§4.3, Fig. 2c–d, Table 1] The comparison of CoV rates between synchronous and round-robin settings is not apples-to-apples. In round-robin, the second mover's Round 1 verdict is already conditioned on the opponent's verdict, so the model's 'conformity' (γ_within) and its 'inertia' (α) can both reflect the same behavior of accepting the prior verdict and then sticking to it. The pooled model in Eq. (2) does not by itself demonstrate that the inertial-in-synchronous / conformist-in-round-robin split is a stable decomposition rather than an artifact of pooling protocols with different information structures. Please report first-round agreement rates separately from later-round revision rates, and ideally fit the model separately for each protocol or include protocol-specific interactions with explicit exposure timing.
minor comments (4)
- [Eq. (2)] The notation nprev_vd and nwithin_vd,r is not fully defined: what counts as a 'previous round' when multiple models are present, and how are counts normalized across rounds of different lengths? Clarify the summation indices and the exact coding for round-robin vs. synchronous settings.
- [Fig. 4 caption] The caption says 'b. The fraction of deliberations where a specific value was inherited,' but the panel structure labels a–c as 'Difference in Value Occurrence' and d–f as inherited values. The text in §4.2 also refers to panels in an order that does not match the caption. Please reconcile.
- [Appendix F] The value list contains 'Environmental consciousness' twice. Deduplicate.
- [Table 1] No model fit statistics (e.g., log-likelihood, AIC, or a null-model comparison) are reported for Eq. (2). Reporting these would help readers judge whether the inertia/conformity decomposition actually improves fit over a model with only baseline and fixed effects.
Circularity Check
No significant circularity: all headline quantities are direct empirical measurements or descriptive model fits, not predictions derived from their own inputs.
full rationale
The paper makes no first-principles derivation or predictive claim that could reduce to its inputs. The design is an observational measurement study: CoV rates, consensus proportions, value Jaccard similarities, and Eq. 2's multinomial parameter estimates are all descriptive summaries of the same logged deliberations. Table 1's α and γ coefficients are estimated from the data and then used to characterize model behavior; because the paper never validates the model on held-out data or calls these 'predictions,' this is standard statistical fitting rather than circularity. The self-citation to Sachdeva & van Nuenen (2025) is used only to situate the AITA corpus and to note that verdict distributions differ from older-model prior work; it does not carry the argument. The Values in the Wild taxonomy is externally sourced and used as a classification scheme, not as evidence for the paper's behavioral findings. The explicitly acknowledged single-run design (Section 5) and absent model-checking diagnostics are validity/reliability concerns, not circularity. The only definitionally-linked observation—that GPT's near-zero CoV rate leaves little room for 'inherited values'—is presented as expected and is not load-bearing for the headline conclusion. Accordingly, no circular step meets the quoted-reduction standard.
Assumptions & free parameters
free parameters (11)
- α_GPT (inertia) =
2.11 (OR 8.27)
- α_Claude (inertia) =
1.50 (OR 4.49)
- α_Gemini (inertia) =
1.04 (OR 2.83)
- γ_prev,GPT =
0.26 (OR 1.29)
- γ_prev,Claude =
0.38 (OR 1.47)
- γ_prev,Gemini =
0.01 (OR 1.01)
- γ_within,GPT =
2.16 (OR 8.68)
- γ_within,Claude =
0.05 (OR 1.05)
- γ_within,Gemini =
1.65 (OR 5.21)
- θ_mv and φ_dv fixed effects =
not reported individually
- V* 48-value subset =
48 values
assumptions (4)
- domain assumption Reddit AITA commenter verdicts are a valid proxy for everyday moral judgments.
- domain assumption Gemini 2.5 Flash value annotations accurately recover the values actually used in explanations.
- ad hoc to paper A single sample at temperature 1 is representative of each model's deliberation behavior.
- standard math The multinomial model with per-dilemma fixed effects and weak L2 regularization yields unbiased estimates of inertia/conformity.
Cite this review
Pith. "Pith review of Interaction Protocol Shapes Moral Judgment in Multi-Agent Debate." pith.science (2026). https://pith.science/paper/HMXKL5V5
@misc{pith2026251010002,
author = {Pith},
title = {Pith review of: Interaction Protocol Shapes Moral Judgment in Multi-Agent Debate},
year = {2026},
howpublished = {\url{https://pith.science/paper/HMXKL5V5}},
note = {Machine review of arXiv:2510.10002}
}
read the original abstract
As large language models (LLMs) are increasingly deployed in sensitive everyday contexts -- offering personal advice, mental health support, and moral guidance -- understanding their behavior in navigating complex moral reasoning is essential. Most evaluations study this sociotechnical alignment through single-turn prompts, but it is unclear if these findings extend to multi-turn settings, and even less clear how they depend on the interaction protocols used to coordinate agentic systems. We address this gap using LLM debate to examine deliberative dynamics and value alignment in multi-turn settings by prompting subsets of three models (GPT-4.1, Claude 3.7 Sonnet, and Gemini 2.0 Flash) to collectively assign blame in 1,000 everyday dilemmas from Reddit's ``Am I the Asshole'' community. To test order effects and assess verdict revision, we use both synchronous (parallel responses) and round-robin (sequential responses) deliberation structures, mirroring how multi-agent systems are increasingly orchestrated in practice. Our findings show striking behavioral differences. In the synchronous setting, GPT-4.1 showed strong inertia (0.6-3.1\% revision rates) while Claude 3.7 Sonnet and Gemini 2.0 Flash were far more flexible (28-41\% revision rates). Value patterns also diverged: GPT-4.1 emphasized personal autonomy and direct communication (relative to its deliberation partners), while Claude 3.7 Sonnet and Gemini 2.0 Flash prioritized empathetic dialogue. We further find that deliberation format had a strong impact on model behavior: GPT-4.1 and Gemini 2.0 Flash stood out as highly conforming relative to Claude 3.7 Sonnet, with their verdict behavior strongly shaped by order effects. We provide additional results on open-source models (DeepSeek-V3.2 and Llama 3.1).
Figures
Figures from the paper (10 more)
Forward citations
Cited by 1 Pith paper
-
Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation
Test-time retrieval of committee-disagreement-mined synthetic images cuts a safety classifier's false-negative rate on a hard HoliSafe subset from 41.2% to 24.5%.
Reference graph
Works this paper leans on
-
[1]
We obtained the 3,272 available AITA submissions from January 1, 2025 to March 30, 2025 using the Reddit API
2025
-
[2]
top-level
For each submission, we obtained the top 100 “top-level” comments (i.e., those that are not replies to other comments). If there were fewer than 100 comments, we obtained all top-level comments
-
[3]
URLhttp://arxiv.org/abs/2406.04214. arXiv:2406.04214. Paul R¨ottger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Sch¨utze, and Dirk Hovy. Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models, June 2024. URL http:// arxiv.org/abs/2402.16786. arXiv:2402.16...
arXiv 2024
-
[4]
edits” or “updates
Reddit posts often contain “edits” or “updates” where the original post provides additional details or responses after their initial submission. For each post, we removed any portion of the text that was an “edit” or “update” using a regular expression
-
[5]
URLhttp://arxiv.org/abs/2410.01639. arXiv:2410.01639 [cs]. Arne Tillmann. Literature review of multi-agent debate for problem-solving, 2025. URL https: //arxiv.org/abs/2506.00066. Anvesh Rao Vijjini, Rakesh R Menon, Jiayi Fu, Shashank Srivastava, and Snigdha Chaturvedi. SocialGaze: Improving the integration of human social norms in large language models. ...
arXiv 2025
-
[6]
doi: 10.18653/v1/2024.findings-emnlp.962
Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.962. URLhttps://aclanthology.org/2024.findings-emnlp.962/. Haotian Wang, Xiyuan Du, Weijiang Yu, Qianglong Chen, Kun Zhu, Zheng Chu, Lian Yan, and Yi Guan. Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate System, July 2024. URLhttp://arxiv.org/abs/2312.048...
arXiv 2024
-
[7]
GPT GPT 0.0 0.1 0.2 0.3 0.4 0.5 Change-of-V erdict Rate
-
[8]
Claude Claude First Second Third Position in Order 0.0 0.1 0.2 0.3 0.4 0.5 Change-of-V erdict Rate
Show all 22 references
-
[9]
Change-of- verdict rates (y-axis) for each model (rows) during round-robin deliberation
Gemini Gemini Figure 10:Change-of-verdict rates during three-way round-robin deliberation. Change-of- verdict rates (y-axis) for each model (rows) during round-robin deliberation. Each experiment is denoted by the x-axis, which indicates what order the specific model went. The...
-
[12]
We identified meta posts either by examining the username or the “flair” attached to the post
We filtered out meta posts, deleted posts, removed posts, or posts that were too short (less than 1,000 characters). We identified meta posts either by examining the username or the “flair” attached to the post
-
[14]
NTA,” “YTA,
We used a regular expression to classify each comment, for each post, as “NTA,” “YTA,” “NAH,” “ESH,” and “INFO.” In cases where we could not cleanly extract a label, we used Gemma-9B to classify the comment
-
[15]
disagreement rate
For each submission, we calculated the proportion of comments assigning each of the five verdicts. We then calculated as “disagreement rate” as the entropy of the verdict proportions
-
[16]
We extracted the top 1,000 dilemmas with the highest disagreements as the final dataset to use for deliberation. D LARGELANGUAGEMODELS We used the following LLMs and corresponding parameters: •GPT-4.1: Versiongpt-4.1-2025-04-14; temperature 1; default parameters • Claude 3.7 S...
2025
-
[17]
These values consisted of clusters grouped together from a more fine-grain list of values
We began with the list of 267 values at the second tier of theValues in the Wildtaxonomy. These values consisted of clusters grouped together from a more fine-grain list of values. We began here in order to control the number of values we classified the deliberation outputs with
-
[18]
We chose the subset of values deemed “moral” by at least 4 of the 5 annotators
We (the authors) and 3 LLM judges (Gemini 2.0 Flash, Claude 3.5 Haiku, GPT-4o) classified all 267 values within four categories:moral,epistemic,aesthetic, andinstrumental. We chose the subset of values deemed “moral” by at least 4 of the 5 annotators. This produced a list of 1...
-
[19]
We then examined the values the consistently appeared across repetitions
Using Gemini 2.5 Flash, we classified model outputs from a random selection of 100 dilemmas 5 separate times, using the list of 110 values. We then examined the values the consistently appeared across repetitions
-
[20]
We then manually considered each value, coding it for inclusion or exclusion from the final set according to the following options 1) inclusion due to relevance and high occurrence in everyday dilemmas; 2) inclusion due to relevance and moderate occurrence in everyday dilemmas...
-
[21]
omit because of overlap with other values; 5) omit due to too large a scope or lack of appearances; 6) omit due to lack of appearances and lack of relevance
-
[22]
Am I the Asshole
We then manually merged 16 values based on their co-occurrences in our trial set to produce the final set of 48 values. F LIST OFVALUES Values are listed in the same randomized order as provided to the LLM judge in a system prompt. • Trust creation and maintenance • Constructi...
-
[2023]
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al
doi: 10.48550/arxiv.2303.17548. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models.arXiv preprint arXiv:2310.13548,...
-
[2024]
arXiv:2410.22932 [cs]
URLhttp://arxiv.org/abs/2410.22932. arXiv:2410.22932 [cs]. Maarten Buyl, Alexander Rogiers, Sander Noels, Iris Dominguez-Catena, Edith Heiter, Raphael Romero, Iman Johary, Alexandru-Cristian Mara, Jefrey Lijffijt, and Tijl De Bie. Large Language Models Reflect the Ideology of ...
2024 arXiv
-
[2025]
ISBN 979-8-89176-251-0
Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/ 2025.acl-long.1210. URLhttps://aclanthology.org/2025.acl-long.1210/. Emanuele La Malfa, Gabriele La Malfa, Samuele Marro, Jie M Zhang, Elizabeth Black, Michael Luck, Philip Torr, and Michael W...
2025
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.