{"id":"2f4d4f78-885b-4207-8fff-f20fdf100b28","arxiv_id":"2412.04937","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Letting the current speaker select the next speaker through adjacency pair rules makes AI agents' group conversations in a murder mystery game more coherent and cooperative.","lead":"This paper builds a multi-agent system where AI characters play a murder mystery game and uses human conversation rules to decide who speaks next. This reduces broken questions and makes AI group discussions more coherent, which matters for games, simulations, and collaborative AI agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CSSN-or-SS vs SS confounds next-speaker selection with explicit '(response)' prompt injection, so the claimed mechanism is not isolated; an ablation is needed.","rationale":"The reader's weakest assumption and my load-bearing concern coincide: the missing ablation between next-speaker selection and the explicit '(response)' prompt constraint. This is the right focus because the paper's design-guideline claim is mechanistic, and the confound is directly testable. The secondary issues, unblinded author-conducted human evaluation and GPT-4 judging conversations produced by GPT-4o-based agents, lower confidence but are not the single decisive issue; they would matter even with a clean ablation. The central empirical finding that the full CSSN-or-SS system outperforms SS may still hold, but the paper's explanation of why it works is underdetermined by the current experiments. Therefore the reader's conditional verdict remains appropriate: the paper should be accepted only with the mechanism isolation and ideally with additional evaluation safeguards.","tokens_in":17961,"tokens_out":3778,"duration_ms":40217,"concrete_test":"Run a fourth condition, CSSN-no-response: identical to CSSN-or-SS (same detectDesignation-based next-speaker selection) but keep the speak() prompt unchanged from SS, i.e., no '(response)' marker and no second-pair-part type. Generate 50 10-turn conversations with the same models and scenario, then compare breakdown counts, LLM-judge scores, and human scores to CSSN-or-SS and SS using the same Dunn tests. If CSSN-no-response matches CSSN-or-SS, next-speaker selection is the active ingredient; if it resembles SS, the '(response)' prompt injection carries the effect. A complementary SS+response condition would verify whether the prompt alone suffices without selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.4's detectDesignation() and Figure 4 show that the CSSN-or-SS condition changes two things relative to SS: (1) it overrides self-selection and chooses the addressee as next speaker, and (2) it rewrites the speak() prompt to include a '(response)' marker plus the second-pair-part type. Section 4.1 defines SS as 'the same as CSSN-or-SS except that it does not have the detectDesignation() mechanism,' so both changes are removed together. Thus the significant reductions in dialogue breakdowns and gains in LLM/human scores can be attributed to the explicit response instruction, to the selection mechanism, or to their interaction; the paper's central claim credits the next-speaker selection system. The discussion's interpretation, e.g., that 'response obligations' suppress inappropriate next speakers and responses to questions, assumes the response obligation comes from selection, but the prompt injection alone could generate the same adjacency-pair behavior even under random or self-selected ordering. This is not a flaw in the system's overall performance, but it is a gap in the evidence for the stated mechanism. The authors acknowledge evaluator subjectivity in Section 6 but do not address this confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MMAgents, a multi-agent framework for LLM-based Murder Mystery game play, which implements two conversation-analysis-inspired turn-taking mechanisms: a self-selection mechanism based on agent-generated importance scores (think()/selectMostImportant()) and a current-speaker-selects-next mechanism (detectDesignation()) that detects adjacency-pair first parts and designates the addressee as next speaker. The authors compare three conditions (EQUAL, SS, CSSN-or-SS) over 50 ten-turn conversations per condition, using dialogue-breakdown analysis, LLM-as-a-judge scoring, and human evaluation. They report that CSSN-or-SS significantly reduces dialogue breakdowns and improves cooperativeness, diversity, information sharing, and reasoning relative to both baselines, and they attribute these gains to the adjacency-pair-based next-speaker selection mechanism.","tokens_in":18176,"tokens_out":3085,"duration_ms":32588,"significance":"If the reported effects are robust and correctly attributed, this is a useful contribution to multi-agent LLM dialogue control: it operationalizes classic conversation-analysis concepts in an implemented system and provides a concrete, statistically tested comparison. The paper is generally clear, the three-condition design is sensible, and the authors are candid about several limitations (e.g., self-conducted human evaluation, topic-control issues). The main weakness is that the CSSN-or-SS condition differs from SS in two coupled ways, so the central mechanism-level claim is not yet isolated.","major_comments":[{"comment":"The comparison between CSSN-or-SS and SS is confounded. In CSSN-or-SS, detectDesignation() both (a) selects the addressee as the next speaker and (b) rewrites the speak() prompt to include an explicit '(response)' marker and the second-pair-part type; SS removes both changes together. The abstract and conclusion attribute the improvements specifically to the next-speaker selection mechanism, but the observed gains could come from the prompt-level response obligation alone, from the selection rule, or from their interaction. An ablation separating these two factors (e.g., SS with the '(response)' prompt injected but without selection, or CSSN-or-SS without the prompt injection) is needed to support the stated mechanism.","section":"§3.2.4, §4.1, Figure 4"},{"comment":"The human evaluation was performed by the authors themselves using a bespoke rubric, with no second annotator, no inter-annotator reliability statistics, and no blinding to condition. Section 6 acknowledges the subjectivity concern, but the conclusion's claims about 'information sharing capabilities and reasoning abilities' depend substantially on this evaluation. Adding at least one independent annotator and reporting agreement (e.g., Cohen's kappa) is necessary to make this evidence load-bearing.","section":"§4.2.3, §6, Figure 7"},{"comment":"The breakdown analysis and LLM-as-a-judge both rely on GPT-4, the same model family used to generate the conversations (GPT-4o for speak() and detectDesignation()). While this does not invalidate the relative comparisons since all conditions are judged by the same model, it introduces a possible same-family bias that is not discussed as a limitation. A brief note in Section 6 about this circularity risk, and ideally a spot-check with a different judge model, would strengthen the evaluation.","section":"§4.2.1–4.2.2, §5"}],"minor_comments":[{"comment":"The Kruskal-Wallis and Dunn tests are reported with p-values, but no effect sizes or medians/interquartile ranges are given for the breakdown counts or judge scores; reporting these would make the magnitude of the improvements more interpretable.","section":"§5"},{"comment":"No accuracy analysis is provided for detectDesignation() (the LLM-based detection of first pair parts and addressee prediction). Since the mechanism's behavior depends on this detector, a small evaluation of its error rate would help readers gauge how noise in detection propagates to the results.","section":"§3.2.4, Figure 5"},{"comment":"The example conversations are shown in English translation from Japanese; the authors note this in a figure caption, but it would be helpful to state explicitly in the text that all prompts and generated utterances were originally in Japanese.","section":"§4.1"},{"comment":"The notation for shortTermHistory uses the same index window length k as History, and the later statement in §4.1 confirms k=5; consider defining k once and noting that the same k is used for both memory types.","section":"§3.1.2, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the core idea is interesting, but the missing ablation is a genuine blocker for the mechanistic claim. I would be willing to review a revised version that adds an ablation separating next-speaker selection from the '(response)' prompt injection, and that either adds independent human annotation or tempers the information-sharing/reasoning claims. The paper could also benefit from a brief discussion of the same-family judge concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nice, focused empirical paper. The concrete contribution is a next-speaker selection mechanism: an LLM detects the first pair part of an adjacency pair, predicts the addressee, and makes that agent the next speaker, while the prompt for that speaker carries an explicit '(response)' marker. The authors layer this on a self-selection importance mechanism, which is a sensible combination. Prior work (AutoGen, Bailis et al., Junprung) hasn't combined adjacency-pair detection with importance-based self-selection. The experiments use 50 ten-turn conversations per condition, significance tests, and clearly chosen example dialogues. The breakdown analysis and the human rubric for information sharing are appropriate for the game domain. Figures 8-10 genuinely show the pattern the authors describe.\n\nThe main soft spot is the confound, which the stress-test note gets right. CSSN-or-SS changes two things at once: who speaks next and what the next speaker's prompt contains. Section 3.2.4 and Figure 4 show the '(response)' constraint and the second-pair-part type are injected into the speak() prompt; Section 4.1 defines SS as identical except without detectDesignation(), so both changes are removed together. The significant reductions in breakdowns and gains in judge/human scores could come from the prompt obligation alone, the selection alone, or the interaction. The conclusion attributes them to the selection system. An ablation - response marker with self-selected ordering, or selection without marker - would settle it. Without it, the mechanism claim is underdetermined, though the practical claim that the combined system works is supported.\n\nTwo smaller concerns. The human evaluation was conducted by the authors; Section 6 acknowledges this but doesn't estimate the bias. And the judge is GPT-4 while the generating model is GPT-4o, so the automatic evaluation isn't fully independent. The non-significant coherence result versus EQUAL is reported correctly, but the conclusion reads like 'comprehensive improvement' despite that.\n\nWho's it for: people building multi-agent dialogue, especially for social deduction or role-play, and anyone working on turn-taking in LLM systems. It's worth a serious referee. The confound should be addressed in revision, but the system, the data, and the clear writing warrant publication.","headline":"A clean empirical study of adjacency-pair turn-taking for multi-agent dialogue; the mechanism claim is confounded by a prompt-injection change, but the system works and deserves a proper peer review.","tokens_in":18702,"tokens_out":2999,"would_cite":true,"duration_ms":29791,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Applying human turn-taking rules to multi-agent LLM dialogue sharply reduces breakdowns and improves cooperation, information sharing, and reasoning in murder-mystery games.","keywords":["turn-taking","conversation analysis","multi-agent systems","LLM agents","adjacency pairs","current speaker selects next","dialogue breakdown","murder mystery"],"falsifier":"Run the same experiment with a fourth condition: select the predicted addressee as next speaker but omit the '(response)' token from their prompt. If breakdown counts and human scores match CSSN-or-SS, the selection mechanism alone is responsible; if they regress toward SS, the prompt-level obligation is the active ingredient.","tokens_in":17753,"feed_emoji":"🗣️","tokens_out":5569,"duration_ms":53695,"temperature":0.7,"pith_summary":"This paper argues that the turn-taking rules conversation analysts have documented in human talk—especially the 'current speaker selects next' technique, where a question or address creates an obligation for a specific addressee to respond—can be transplanted into multi-agent LLM dialogue to make it more orderly and effective. The authors build 'Murder Mystery Agents,' a framework in which an LLM detects when an utterance is the first half of an adjacency pair (such as a question) and selects the named addressee as the next speaker, adding an explicit '(response)' constraint to that agent's prompt. In a four-player murder mystery discussion, this condition produced significantly fewer dialogue breakdowns, higher judged cooperation and diversity, and better information sharing and reasoning scores than both random turn assignment and pure self-selection. The work matters because it turns a descriptive science of human conversation into a design rule for AI systems, and it gives a concrete, cheap mechanism—detect the adjacency pair, assign the obligation—that other multi-party agent architectures could adopt.","feed_headline":"Conversation rule from human talk tames AI group chat","feed_subtitle":"Adding 'current speaker selects next' to LLM agents lowers dialogue breakdowns and boosts info sharing and reasoning.","key_machinery":"The central mechanism is the detectDesignation() module coupled with the CSSN-or-SS turn-taking rule. detectDesignation() uses an LLM to classify whether the latest utterance is the first pair part of an adjacency pair (a yes/no question, a wh-question, or a direct address) and to predict the addressee; that addressee becomes the next speaker, and the prompt for the next utterance carries an explicit '(response)' constraint. This enforces the 'current speaker selects next' rule from conversation analysis, so that a question is immediately followed by the answer from the person asked, rather than by a self-selected speaker or a random turn.","core_discovery":"On the authors' own terms, the discovery is that imposing response obligations derived from adjacency pairs makes LLM-based multi-party conversation work better. When an utterance contains a first pair part (a question or a direct address), the detectDesignation() module identifies the type of the pair and predicts the intended next speaker; that speaker's next-generation prompt is then marked with '(response)'. Across 50 ten-turn conversations per condition, this CSSN-or-SS condition reduced dialogue-breakdown utterances by a wide margin compared with EQUAL random turns and SS self-selection, with the Kruskal–Wallis test showing significant differences for breakdowns, cooperation, diversity, and the human murder-mystery criteria. The authors read this as support for the theoretical claim that sequence organization is the source of conversational coherence, now evidenced inside machine-machine dialogue.","pith_inferences":["A natural extension the paper leaves implicit is to use the same adjacency-pair detection to manage topic length: when a sequence is closed with a sequence-closing third such as 'I see, thank you', the system could score the topic as finished and nudge agents toward new information.","Because the paper reports long-term memory degradation around 30 turns, the observed turn-taking benefit may shrink in longer dialogues; a direct test is to plot breakdown frequency against turn number and see whether CSSN-or-SS loses its edge.","The mechanism's strategic-evasion behavior (partial answers, topic shifts) suggests that response obligations are weighed against character goals; this could be repurposed to create agents with calibrated secrecy—obliged to respond, but free to withhold the core secret.","One could also swap the LLM-based detector for a simpler rule-based one in constrained domains; if the gains are mostly from the response constraint, rule-based detection may be sufficient, making the approach even cheaper."],"forward_implications":["Multi-agent LLM systems can reduce dialogue breakdowns by routing turns through the addressee detected in the previous utterance.","In collaborative reasoning games, structuring turns by adjacency pairs leads agents to share more mission-relevant facts and to reason more logically, per the paper's human evaluation.","The mechanism is cheap: one extra LLM call per turn to detect the first pair part, with no retraining or fine-tuning.","The design guidance generalizes beyond games: any chat platform with multiple AI participants could adopt the same obligation-based turn-taking.","The result supports the theoretical claim that sequence organization (adjacency pairs) is a source of conversational coherence, now demonstrated empirically inside machine-machine talk."],"supporting_citations":[{"why":"Supplies the turn-taking rules (current speaker selects next, self-selection) that the framework implements.","marker":"[20]"},{"why":"Defines adjacency pairs, the basic unit of utterance sequences that the response-obligation mechanism relies on.","marker":"[18]"},{"why":"Formulates conditional relevance, the normative binding that the '(response)' constraint instantiates.","marker":"[19]"},{"why":"Provides the theoretical prediction that sequence organization yields conversational coherence, which the experiment tests.","marker":"[21]"},{"why":"Provides the taxonomy of utterances that lead to dialogue breakdowns, used as the primary outcome measure.","marker":"[39]"},{"why":"Motivates the need for dynamic turn-taking and is the contrast point for autonomy; the paper positions its mechanism against this prior approach.","marker":"[17]"}],"fun_headline_variants":["Human turn-taking rules improve AI group conversations","Adjacency pairs curb AI dialogue breakdowns","Who speaks next? AI learns from murder mystery chat","LLM agents chat smoother with speaking-order rules","Conversation analysis helps AI agents cooperate better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never tests next-speaker selection without the explicit '(response)' note in the prompt, so the improvement could come from that nudge rather than from choosing the addressee as the next speaker.","fun_headline_variants_meta":{"raw":{"variants":["Human turn-taking rules improve AI group conversations","Adjacency pairs curb AI dialogue breakdowns","Who speaks next? AI learns from murder mystery chat","LLM agents chat smoother with speaking-order rules","Conversation analysis helps AI agents cooperate better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000417,"raw_usage":{"total_tokens":2161,"prompt_tokens":965,"completion_tokens":1196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1127}},"tokens_in":581,"tokens_out":1196,"duration_ms":12745,"temperature":1.0,"reasoning_tokens":1127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:06:36.462587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiment with a fourth condition: select the predicted addressee as next speaker but omit the '(response)' token from their prompt. If breakdown counts and human scores match CSSN-or-SS, the selection mechanism alone is responsible; if they regress toward SS, the prompt-level obligation is the active ingredient.","supporting_citations":[{"cited_title":"Higashinaka, M","cited_arxiv_id":null,"evidence_quote":"Provides the taxonomy of utterances that lead to dialogue breakdowns, used as the primary outcome measure."},{"cited_title":"Sacks, E","cited_arxiv_id":null,"evidence_quote":"Supplies the turn-taking rules (current speaker selects next, self-selection) that the framework implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines adjacency pairs, the basic unit of utterance sequences that the response-obligation mechanism relies on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Formulates conditional relevance, the normative binding that the '(response)' constraint instantiates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the theoretical prediction that sequence organization yields conversational coherence, which the experiment tests."}],"review_version":1}