Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

Herd Behavior: Investigating Peer Influence in LLM-based Multi-Agent Systems

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A confident peer answer flips LLM agents' choices 48% of the time, and presentation format can tune the effect.

desk verdict Solid empirical study with a confounded central claim: the confidence-gap result needs an independent manipulation of peer confidence, but the format, order, and controllability findings are worth keeping. read the letter →

arxiv 2505.21588 v1 pith:MTR2UBZK submitted 2025-05-27 cs.MA cs.AI

Young-Min Cho , Sharath Chandra Guntuku , Lyle Ungar This is my paper · ORCID
classification cs.MAcs.AI
keywords herdbehaviormulti-agentsystemspeerinfluenceconfidenceconformitypresentationformatflipratelargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language model agents, when shown one another's answers in a multi-agent setting, behave like a herd—changing their own answers toward peers—and what controls that tendency. Across controlled experiments on six benchmarks (three factual, three opinionated), the central finding is that an agent's likelihood of flipping its answer is driven by the gap between its self-confidence and the perceived confidence of the peer answer: the larger the gap, the stronger the herding. The most persuasive peer answer (the model's own second-most-probable response) produces an average flip rate of 0.48, yet on factual questions this conformity lowers accuracy compared with the agent's original answer. Presentation format and order are also powerful modulators: counts and percentages make agents resist the crowd, listing individual answers with reasons amplifies conformity, and showing disagreement before agreement increases flips. The paper concludes that herd behavior is not fixed but can be systematically tuned, and that structural presentation cues are far more effective control levers than prompt-level instructions such as 'be agreeable'.

What carries the argument

The machinery is the flip-rate measurement: a binary indicator $I_{\text{flip}}(a_i, q_k)$ that equals 1 if an agent changes its response after observing a peer's answer, averaged over questions and agents. Confidence is operationalized directly from the model's token-level probability $P(r \mid C)$ over the multiple-choice options, with self-confidence read as the probability of the agent's own first choice and perceived peer confidence as the probability the same distribution assigns to the peer's presented answer. The pivotal device is the '2nd' peer condition, which presents the agent's own second-most-probable response as the peer's answer; this is treated as the most persuasive alternative and reliably produces the strongest herding. The format experiments then layer five presentation methods (Count, Ratio, List, Disc, Reason) and two orderings (agree-first vs. disagree-first) onto the same flip-rate signal to show how structural cues modulate conformity.

What would settle it

Run the same two-agent protocol with peer answers supplied by a different model or by human-written responses, so the 'peer' is no longer drawn from the target agent's own token distribution. If flip rates no longer track the gap between self-confidence and perceived peer confidence, the measured herding is an artifact of shared model statistics rather than a genuine social dynamic.

Watch

Extended reading notes

Core claim

The paper's central claim is that herd behavior in LLM-based multi-agent systems is primarily driven by the relationship between an agent's self-confidence and its perceived confidence in peers: lower self-confidence combined with higher perceived peer confidence significantly increases the likelihood of conformity. In a minimal two-agent protocol, the authors sample a peer's answer from the target agent's own response distribution—first, second, random, or least probable—and find the second-most-probable response ('2nd') consistently yields the highest flip rates across benchmarks (0.48 average), while also causing a statistically significant drop in accuracy on factual tasks such as MMLU-Pro and ARC-Challenge. They further show that presentation format modulates herding: Count and Ratio summaries produce a sharp boundary at the diagonal where agree and disagree counts are equal, with low flip rates when agreement dominates, whereas List, Disc, and Reason formats show strong herding mainly when few agents agree; Reason yields the highest overall flip rates. Ordering also matters: presenting disagreement before agreement amplifies conformity relative to the reverse, despite identical information. Finally, combining amplifying factors (graduate-degree persona, Reason format, disagreement-first) yields the highest flip rate (0.63) and the best group accuracy on MMLU-Pro (0.29, above chain-of-thought), while 'be agreeable'/'be stubborn' prompt controls barely move the flip rate, leading the authors to describe the agents as boundedly rational: responsive to confidence signals and social cues, but vulnerable to framing.

Load-bearing premise

The paper's load-bearing assumption is that the model's own probability score for a peer's answer is a faithful measure of how confident that peer appears; if this proxy is wrong, the confidence-gap mechanism could be an artifact of prompt construction rather than genuine social influence.

Editorial extensions

If this is right

  • If the confidence-gap result holds, multi-agent systems can dampen herd flips simply by presenting peer answers as summary counts or percentages rather than as individual voices.
  • Placing the most persuasive alternative (the second-most-probable answer) among peer responses is a reliable way to induce flips, but on factual benchmarks it measurably degrades accuracy, so its use should be limited to tasks where convergence itself is the goal.
  • Ordering peer information is an effective, low-cost control: leading with disagreement amplifies herding, leading with agreement suppresses it, even with identical content.
  • The Strong Factors recipe (authoritative persona, reasons, disagreement-first) outperforms chain-of-thought on MMLU-Pro group accuracy, suggesting that carefully engineered peer input can improve collective performance rather than merely corrupt it.
  • Because prompt-level instructions ('be agreeable'/'be stubborn') have minimal effect compared with structural factors, system designers should expect meta-instructions to be an unreliable governance tool in multi-agent LLM systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the same framing effects likely transfer to human-facing AI interfaces: how a system aggregates and displays model opinions (counts vs. reasons, order of arguments) could shape human users' conformity just as it shapes agent flips.
  • The paper's two-agent design isolates a single social signal; a natural extension would test whether the confidence-gap rule predicts cascade depth in larger agent societies (e.g., 10+ agents), where information cascades are the collective outcome.
  • Because peer answers are sampled from the target agent's own distribution, the 'perceived confidence' measure is partially a statement about the agent's priors; testing with out-of-distribution peer answers (different model, human crowd) would separate social influence from statistical self-consistency.
  • The weak effect of prompt-based control hints at a broader governance problem: in black-box LLM systems, high-level instructions are unreliable, so designers must manipulate observed content rather than instruct behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports controlled experiments on herd behavior in LLM-based multi-agent systems, using multiple-choice benchmarks and GPT family models. It defines self-confidence and perceived peer confidence from the target model's own token probabilities, manipulates the peer answer type (1st, 2nd, random, last), peer persona, presentation format, and presentation order, and measures flip rates, entropy, consensus, and accuracy. The central claims are that flip rates peak when self-confidence is low and perceived peer confidence is high; that the '2nd' peer answer is the most persuasive; that presentation format and order modulate herding, with disagreement-first and reason-based formats amplifying it; and that structural factors control herding more effectively than prompt-level instructions. The paper also reports accuracy costs of herding on factual benchmarks.

Significance. If the central result holds, the paper would provide a useful empirical map of when and how LLM agents conform to peers, with design implications for multi-agent collaboration systems. The work has genuine strengths: it tests six benchmarks spanning factual and opinionated tasks, includes rnd and last conditions as controls, reports results across five model variants in Appendix A, and makes transparent, falsifiable definitions of flip rate and confidence. The comparison between structural presentation factors and prompt-based interventions is a valuable contribution. However, the perceived-confidence manipulation is not independent of the target agent's own response distribution, so the headline mechanistic claim about a 'confidence gap' is not yet separated from a near-tie artifact. The lack of code/data, uncorrected multiple comparisons, and missing confidence intervals further weaken the empirical support.

major comments (3)
  1. [Section 3.1 and Figure 2] The operationalization of perceived peer confidence is confounded with self-confidence. Because the peer answer r_j (especially the '2nd' condition) is selected from the target model's own distribution P(r|q), the cell labeled 'low self-confidence / high perceived confidence' is also the cell in which P(r_i|q) and P(r_j|q) are nearly equal; under greedy argmax decoding, any context that slightly favors r_j will flip the choice. The high flip rates in that cell could therefore be a near-tie artifact of the stimulus construction rather than evidence that agents defer to a confident peer. The Limitations section concedes that perceived confidence is a static proxy but does not address this base-rate confound. To support the central claim, the paper needs at least one of: (i) a control condition with a neutral non-answer peer message on the same low-confidence questions, (ii) flip rates stratified by the logit or probability gap between r_i and r_j, or (iii) an independent manipulation of peer confidence (e.g., explicit confidence statements) that is orthogonal to P(r_j|q). Without such an analysis, the abstract and Section 3.3 claim that the confidence gap drives herding is not established.
  2. [Tables 2, 3, 5, 6] The paper reports many paired t-tests at the p<0.05 level within each group without any multiple-comparison correction. Across six benchmarks, five models, and numerous persona and presentation conditions, dozens of comparisons are made, so some nominally significant differences would be expected by chance. This is load-bearing for the claim that the 2nd condition is consistently the most persuasive and that persona effects are significant. Please report adjusted p-values or confidence intervals, and include effect sizes for the headline 2nd-versus-last comparisons.
  3. [Abstract footnote and Appendix A/B] Reproducibility is currently insufficient. The abstract states 'Code and data will be released in the camera-ready,' but no code or data accompanies the submission, and the exact sampling procedure, number of repeated trials, random seeds, and API call details are not fully specified. Since the entire paper is an empirical measurement of flip rates, the central numbers in Figures 2-4 and Tables 2-6 cannot be independently checked. Please make the code/data available with the submission or provide a complete reproducibility appendix with per-question results.
minor comments (5)
  1. [Figure 2] The heatmap has no colorbar, no error bars or confidence intervals, and the binning procedure for self-confidence and perceived confidence is not described. Please specify the bin edges and report variability.
  2. [Section 5.2] The diversity filter threshold (maximum original-response probability less than 0.8) is an arbitrary free parameter; the authors should report how the results in Table 3 change under nearby thresholds.
  3. [Section 3.3 and Table 2] The text states that 'employer as peer caused weaker herd behavior in opinionated benchmarks,' but the differences in Table 2 are small (0.71 vs. 0.74 on OpinionQA and 0.77 vs. 0.79 on GlobalOpinionQA) and are not marked as significant; the wording overstates the effect.
  4. [Appendix A] The sentence 'responses positioned 2nd consistently exhibit higher flip rates than those in the last position, suggesting a greater susceptibility to herd behavior when confidence is perceived earlier' confuses the 2nd response type with presentation order. The 2nd condition is a probability-ordered response type, not an ordering manipulation; please rephrase.
  5. [Table 4] The Pearson correlations are reported as significant at p<0.001, but with thousands of questions even negligible correlations will be significant. Please report the correlation coefficients with confidence intervals and interpret the effect sizes.

Circularity Check

1 steps flagged · score 6.0 of 10

The Figure 2 confidence-gap result is partly built into the stimulus construction: peer confidence is read off the target model's own token distribution, so the 'most persuading peer answer' is by definition the second-most-probable answer.

  1. self definitional [Section 3.1, Figure 2, Table 2; abstract]
    "From the agent's original distribution P (r | q) over possible responses to question q, we manually select one of four types of responses to serve as the peer's opinion rj: ... 2nd: The second most probable response, chosen to represent a highly persuasive alternative from the agent's perspective. ... We then compute the flip rate across all questions, analyzing how the strength of herd behavior related with the agent's self-confidence P (ri | q) and the perceived confidence P (rj | q)."

    Both Figure 2 axes come from the same P(r|q): self-confidence is P(ri|q) (max) and perceived confidence is P(rj|q), where rj is chosen as 2nd/rnd/last from that same distribution. So the peer carries no independent confidence signal; in the 2nd condition, 'high perceived confidence' just means the second-most-likely answer is close in probability to the first. Low self-confidence flattens the distribution, making the gap small by construction and the agent near-indifferent; high flip rates in that cell can be a near-tie artifact of argmax selection, not social influence. The abstract's 'most persuading peer answer driving the strongest herding' restates that the second-highest-probability answer is the most probable alternative.

full rationale

The paper's other principal results are not circular: the presentation-format experiments (Count, Ratio, List, Disc, Reason), the order manipulation, and the persona manipulations (education, hierarchy, domain expertise) vary externally controlled prompt content and are evaluated on standard external benchmarks (MMLU-Pro, GPQA-Diamond, ARC-Challenge, etc.). Those comparisons do not reduce to the model's own token probabilities. The citation of the authors' prior work (Cho et al., 2024) is contextual and not load-bearing. However, the paper's first and headline contribution--that the gap between self-confidence and perceived confidence drives herding--is confounded by stimulus construction: rj is sampled from the same P(r|q) used to define self-confidence, and Figure 2 bins flip rates against two non-independent quantities from that single distribution. The Limitations section concedes that perceived confidence is a 'static representation' but does not address this base-rate/near-tie confound. Because the confidence-gap claim is central but the format, order, persona, and controllability results retain independent content, the circularity is partial rather than total.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No fitted constants or new physical mechanisms are introduced. The only hand-set selection is the 0.8 probability filter in Section 5.2. The paper's main assumptions are modeling choices about how to read confidence and flipping from LLM token probabilities.

free parameters (1)
  • Diversity filter threshold (max original-response probability) = 0.8
    Section 5.2 keeps only questions where the highest probability among original responses is below 0.8. This hand-chosen threshold ensures response diversity but also biases the sample toward contentious questions, and it directly shapes the Section 5 accuracy and flip-rate results.
assumptions (3)
  • domain assumption Token-level probability P(r|C) is a valid measure of an LLM agent's confidence in a candidate answer.
    Invoked in Definition 1 and Section 3.1; cited to Xiao and Wang 2019, but treated as a faithful proxy for confidence even though the model is not calibrated in the usual sense.
  • domain assumption A flip between the initial and revised answer is a valid operationalization of herd behavior.
    Definition 3 adopts Laban et al. 2023; the entire flip-rate metric depends on this reduction of herding to a binary answer switch on multiple-choice items.
  • ad hoc to paper The second-most-probable response under the target agent's own distribution is the most persuasive alternative.
    Section 3.1 defines the 2nd peer condition this way; this construction is specific to this paper and directly controls the observed flip rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Herd Behavior: Investigating Peer Influence in LLM-based Multi-Agent Systems." pith.science (2026). https://pith.science/paper/MTR2UBZK

@misc{pith2026250521588,
  author       = {Pith},
  title        = {Pith review of: Herd Behavior: Investigating Peer Influence in LLM-based Multi-Agent Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MTR2UBZK}},
  note         = {Machine review of arXiv:2505.21588}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) have enabled the emergence of multi-agent systems where LLMs interact, collaborate, and make decisions in shared environments. While individual model behavior has been extensively studied, the dynamics of peer influence in such systems remain underexplored. In this paper, we investigate herd behavior, the tendency of agents to align their outputs with those of their peers, within LLM-based multi-agent interactions. We present a series of controlled experiments that reveal how herd behaviors are shaped by multiple factors. First, we show that the gap between self-confidence and perceived confidence in peers significantly impacts an agent's likelihood to conform. Second, we find that the format in which peer information is presented plays a critical role in modulating the strength of herd behavior. Finally, we demonstrate that the degree of herd behavior can be systematically controlled, and that appropriately calibrated herd tendencies can enhance collaborative outcomes. These findings offer new insights into the social dynamics of LLM-based systems and open pathways for designing more effective and adaptive multi-agent collaboration frameworks.

Figures

Figures reproduced from arXiv: 2505.21588 by the authors.

Figure 1
Figure 1. An example of herd behavior: Even when un [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Flip rate across varying levels of self [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of average flip rates across five presentation formats. Each heatmap shows the average [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of average flip rates across two [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Open-ended LLM conformity is not captured by answer flips: wrong peers degrade revisions, and judges' ratings shift when peer context is visible, so evaluation must be modeled explicitly.

  2. When Truth Is Distributed: Misinformation Derails Collective Fact Recovery in LLM-Based Multi-Agent Systems

    cs.MA 2026-08 conditional novelty 7.0 of 10

    A single deceptive evidence holder in an LLM multi-agent system sharply reduces collective fact recovery, and the false account propagates through honest agents.

  3. Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarks

    cs.CL 2026-07 accept novelty 7.0 of 10

    Across six open-weight LLMs and seven datasets, a speaker-free wrong-answer assertion alone flips 66.5% of initially correct answers, versus 10.3% for a plain re-ask; source labels mainly add a modest increment above ...

  4. Probing Multimodal Large Language Models on Cognitive Biases in Chinese Short-Video Misinformation

    cs.CL 2026-01 unverdicted novelty 7.0 of 10

    Multimodal LLMs exhibit different levels of susceptibility to misinformation in short videos, with Gemini-2.5-Pro showing the highest resistance (belief score 71.5) and o3 the lowest (35.2).

  5. Social Networks of LLM Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Attention width and source social power determine whether LLM agent networks herd or achieve wisdom-of-crowds, with a pricing equalizer restoring optimal collective weights.

  6. MAD-Spear: A Conformity-Driven Prompt Injection Attack on Multi-Agent Debate Systems

    cs.CR 2025-07 conditional novelty 6.0 of 10

    MAD-Spear is a prompt injection attack that makes a single compromised agent emit fake 'Sybil' peer answers, exploiting LLM conformity to steer a multi-agent debate toward a wrong consensus and higher token costs.

Reference graph

Works this paper leans on

39 extracted references · 16 canonical work pages · cited by 6 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Solomon E Asch. 2016. Effects of group pressure upon the modification and distortion of judgments. In Organizational influence processes, pages 295--303. Routledge

    2016

  3. [3]

    Abhijit V Banerjee. 1992. A simple model of herd behavior. The quarterly journal of economics, 107(3):797--817

    work page 1992

  4. [4]

    Dan Bang, Laurence Aitchison, Rani Moran, Santiago Herce Castanon, Banafsheh Rafiee, Ali Mahmoodi, Jennifer YF Lau, Peter E Latham, Bahador Bahrami, and Christopher Summerfield. 2017. Confidence matching in group decision-making. Nature Human Behaviour, 1(6):0117

    work page 2017

  5. [5]

    Sushil Bikhchandani, David Hirshleifer, and Ivo Welch. 1992. A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of political Economy, 100(5):992--1026

    work page 1992

  6. [6]

    Nuo Chen, Jiqun Liu, Xiaoyu Dong, Qijiong Liu, Tetsuya Sakai, and Xiao-Ming Wu. 2024. Ai can be cognitively biased: An exploratory study on threshold priming in llm-based batch relevance assessment. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 54--63

    work page 2024

  7. [7]

    Young-Min Cho, Raphael Shu, Nilaksh Das, Tamer Alkhouli, Yi-An Lai, Jason Cai, Monica Sunkara, and Yi Zhang. 2024. Roundtable: Investigating group decision-making mechanism in multi-agent collaboration. arXiv preprint arXiv:2411.07161

    work page Pith review arXiv 2024

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

Show all 39 references
  1. [9]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning

    2023

  2. [10]

    Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2023. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXi...

    2023 arXiv

  3. [11]

    Liye Fu, Lillian Lee, and Cristian Danescu-Niculescu-Mizil. 2017. When confidence and competence collide: Effects on online decision-making discussions. In Proceedings of the 26th international conference on world wide web, pages 1381--1390

    2017

  4. [12]

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680

    2024 arXiv

  5. [13]

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 3(4):6

    2023 arXiv

  6. [14]

    Jaap Jumelet, Willem Zuidema, and Arabella Sinclair. 2024. Do language models exhibit human-like structural priming effects? arXiv preprint arXiv:2406.04847

    2024 arXiv

  7. [15]

    Philippe Laban, Lidiya Murakhovs' ka, Caiming Xiong, and Chien-Sheng Wu. 2023. Are you sure? challenging llms leads to performance drops in the flipflop experiment. arXiv preprint arXiv:2311.08596

    2023 arXiv

  8. [16]

    Tongxuan Liu, Xingyu Wang, Weizhe Huang, Wenjiang Xu, Yuting Zeng, Lei Jiang, Hailong Yang, and Jing Li. 2024 a . Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion. arXiv preprint arXiv:2409.14051

    2024

  9. [17]

    Xuan Liu, Jie Zhang, Haoyang Shang, Song Guo, Chengxu Yang, and Quanyan Zhu. 2024 b . Exploring prosocial irrationality for llm agents: A social cognition view. arXiv preprint arXiv:2405.14744

    2024 arXiv

  10. [18]

    d, Juliane E K \

    Mehdi Moussa \" d, Juliane E K \"a mmer, Pantelis P Analytis, and Hansj \"o rg Neth. 2013. Social influence and the collective dynamics of opinion formation. PloS one, 8(11):e78433

    2013

  11. [19]

    Lev Muchnik, Sinan Aral, and Sean J Taylor. 2013. Social influence bias: A randomized experiment. Science, 341(6146):647--651

    2013

  12. [20]

    OpenAI. 2023. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o-mini: Advancing cost-efficient intelligence . Accessed: 2024-08-18

    2023

  13. [21]

    OpenAI . 2024. GPT-4.1 . https://openai.com/index/gpt-4-1/. Accessed: 2025-05-20

    2024

  14. [22]

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1--22

    2023

  15. [23]

    Niccol \`o Pescetelli, Anna-Katharina Hauperich, and Nick Yeung. 2021. Confidence, advice seeking and changes of mind in decision making. Cognition, 215:104810

    2021

  16. [24]

    Ramsey M Raafat, Nick Chater, and Chris Frith. 2009. Herding in humans. Trends in cognitive sciences, 13(10):420--428

    2009

  17. [25]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  18. [26]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971--30004. PMLR

    2023

  19. [27]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728

    2019 arXiv

  20. [28]

    Jonathan Shaki, Sarit Kraus, and Michael Wooldridge. 2023. Cognitive effects in large language models. In ECAI 2023, pages 2105--2112. IOS Press

    2023

  21. [29]

    Arabella Sinclair, Jaap Jumelet, Willem Zuidema, and Raquel Fern \'a ndez. 2022. Structural persistence in language models: Priming as a window into abstract language representations. Transactions of the Association for Computational Linguistics, 10:1031--1050

    2022

  22. [30]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Informat...

    2024

  23. [31]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  24. [32]

    Zhiyuan Weng, Guikun Chen, and Wenguan Wang. 2025. Do as we do, not as you think: the conformity of large language models. arXiv preprint arXiv:2501.13381

    2025 arXiv

  25. [33]

    Zengqing Wu and Takayuki Ito. 2025. The hidden strength of disagreement: Unraveling the consensus-diversity tradeoff in adaptive multi-agent systems. arXiv preprint arXiv:2502.16565

    2025 arXiv

  26. [34]

    Yijun Xiao and William Yang Wang. 2019. Quantifying uncertainties in natural language processing tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7322--7329

    2019

  27. [35]

    Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2023. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940

    2023 arXiv

  28. [36]

    Paul Zarnoth and Janet A Sniezek. 1997. The social influence of confidence in group decision making. Journal of Experimental Social Psychology, 33(4):345--366

    1997

  29. [37]

    Xiaochen Zhu, Caiqi Zhang, Tom Stafford, Nigel Collier, and Andreas Vlachos. 2024. Conformity in large language models. arXiv preprint arXiv:2410.12428

    2024 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.