Pith. sign in

REVIEW 4 major objections 5 minor 10 references

Silicon Minds versus Human Hearts: The Wisdom of Crowds Beats the Wisdom of AI in Emotion Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPT-4o reads emotions better than any single human, but a crowd of humans beats a crowd of GPT-4o—and mixing them is best.

desk verdict A competent empirical study whose genuinely new crowd-voting result is directionally credible but statistically fragile; the central reversal likely holds, but the augmented-intelligence edge is near-ceiling noise. read the letter →

arxiv 2508.08830 v1 pith:PZSXPLNC submitted 2025-08-12 cs.AI cs.CVcs.CY

Mustafa Akben , Vinayaka Gude , Haya Ajjan This is my paper · ORCID
classification cs.AIcs.CVcs.CY
keywords emotionrecognitionmultimodallargelanguagemodelsGPT-4owisdomofthecrowdpluralityvotingaugmentedintelligenceRMETMRMET
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the individual-level superiority of GPT-4o—a model that processes both images and text—survives the transition from one judge to many judges. On two widely used tests of reading emotion from photos of eyes, GPT-4o scores above the human average and even above most expert-level humans, with first- and second-order stochastic dominance over the human score distribution; the authors caution that the original RMET images may have been in GPT-4o's training data, making the newer MRMET the cleaner test. Yet when individual answers are combined by plurality voting into crowds, human crowds improve steeply with group size while GPT-4o crowds stay nearly flat, reversing the individual result (predicted accuracy 0.998 vs 0.959 on the original Eyes Test). The mechanism is that repeated sampling of one model is like asking one person repeatedly, so errors repeat instead of canceling. A 90:10 human–AI mix outperforms either crowd alone, indicating that independent human diversity and AI consistency are complementary rather than competing.

What carries the argument

Plurality-voting aggregation: for a given test item, each member of a randomly composed group casts one vote for one of the four emotion labels, and the label with the most votes becomes the group's answer; groups range from n = 5 to n = 90 and are drawn repeatedly with replacement. The crowd-level analysis is a mixed-effects logistic regression of item correctness on condition (human, GPT-4o, human+GPT-4o) and log10 group size, with random intercepts for items and iterations. The load-bearing contrast is the interaction between condition and log group size: human crowds gain accuracy steeply with size (slope 3.99 on RMET) while GPT-4o crowds barely move (slope 0.25), which is the formal sig

What would settle it

Re-run the crowd-level mixed-effects model with the resampling design respected—for example, cluster-robust standard errors by the underlying human participant and GPT-4o run, or a permutation test drawing groups without replacement—and ask whether the human-vs-GPT-4o crowd contrast on RMET (reported OR = 26.0) remains large; a collapse toward 1 would mean the reversal is an artifact of pseudoreplication rather than a genuine wisdom-of-crowds effect.

Watch

Extended reading notes

Core claim

On the Reading the Mind in the Eyes Test (RMET) and its multiracial counterpart (MRMET), the paper reports that GPT-4o outperforms humans both on average (0.90 vs 0.71 and 0.83 vs 0.62) and across the full performance distribution, including the 97th percentile. The new result is the crowd-level reversal: when individual responses are pooled by plurality voting into groups of 5 to 90 members, human crowds approach near-perfect accuracy as group size grows (0.998 on RMET, 0.960 on MRMET), while GPT-4o crowds plateau around 0.959 and 0.924, so human crowds beat AI crowds by odds ratios of 26.0 and 1.95 respectively. The mechanism is that the 100 GPT-4o runs are near-deterministic: every API ca

Load-bearing premise

The crowd-level reversal rests on treating each of the 6,600 resampled groups per dataset as an independent observation even though the same 17,680 human answer sheets and the same 100 GPT-4o runs are recycled across thousands of groups—if that independence fails, the reported crowd gap and its confidence intervals shrink—and although the authors separately acknowledge a possible RMET contamination of GPT-4o's training data, that dependence is not modeled.

Editorial extensions

If this is right

  • If the crowd-level reversal is correct, evaluations claiming AI matches human social intelligence should include aggregated judgments, not just single-model accuracy, because individual and collective results can point in opposite directions.
  • In high-stakes emotion-recognition settings such as clinical assessment or security screening, multiple independent human raters would outperform many calls to the same model, since the model's errors do not average away.
  • The augmented-intelligence result implies a practical deployment pattern: collect independent human and model judgments and combine them by majority vote, rather than letting humans interact with model suggestions directly.
  • For model developers, the flat AI crowd slope suggests that sampling many outputs from one set of weights is not a substitute for diversity across models; error cancellation requires genuinely different predictors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicit in these results but not directly tested: the 90:10 mixing ratio is a single point on a continuum; varying the ratio and the number of distinct models would show when AI votes stop adding independent information.
  • The flat GPT-4o crowd slope suggests an 'effective number of independent voters' near 1 for a single model; if that generalizes, model evaluation should report this quantity, not just accuracy.
  • The same independent-vote aggregation design is portable to other diagnostic tasks—medical image reading, credibility assessment—where direct human–AI interaction has known biases, and pooled votes could be compared against interacting teams.
  • Because the authors note possible RMET contamination, a direct test is to run GPT-4o on a freshly created Eyes-test-style item set; if the individual-level RMET advantage shrinks to the MRMET level, part of the individual superiority is memorization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper compares GPT-4o with human performance on two emotion-recognition tests (RMET, MRMET) using public human datasets and 100 GPT-4o API runs. It reports three main findings: (RQ1) GPT-4o outperforms humans at the individual level across the whole performance distribution, (RQ2) after plurality voting, human crowds outperform GPT-4o crowds, and (RQ3) an augmented crowd of 90% human and 10% AI votes achieves the highest accuracy. The analysis uses mixed-effects logistic regression, bootstrap shift functions, first- and second-order stochastic dominance tests, and a resampling-based plurality-voting procedure for groups of size 5 to 90.

Significance. If the claims were fully substantiated, the reversal from individual AI superiority to collective human superiority would be an important and practically relevant result for human-AI collaboration, with implications for high-stakes emotion-recognition deployment. The paper has clear strengths: it uses two standardized tests, transparently discusses dataset contamination, and provides detailed descriptions of the statistical procedures. However, several load-bearing points currently lack support: the key mechanism for the AI-crowd plateau (item-level consistency of GPT-4o runs) is asserted but not measured; the stochastic-dominance inference is based on an invalid interpretation of non-significance as evidence; and the augmented-intelligence result depends on an arbitrary mixing ratio and near-ceiling differences. These issues are fixable within the scope of the manuscript, but they require new analyses or substantial reframing.

major comments (4)
  1. [§2.3, §4 (Plurality Voting; Discussion)] The central explanation for the AI-crowd plateau is that repeated GPT-4o sampling 'resembles asking one person repeatedly,' with the manuscript stating that 'across all test items, the model's choices remained consistent regardless of sampling frequency.' This is an empirical claim about item-level response diversity, but no item-level evidence is provided. Table 1 reports only mean-accuracy SD across runs, which does not rule out large item-level disagreements. Since AI groups are formed by drawing with replacement from only 100 response vectors and can reach size 90, the flat AI-crowd curve in Figure 3 could be an artifact of the finite, heavily overlapping pool rather than an intrinsic property of GPT-4o. Please report per-item response entropy, pairwise agreement, or a clustering/partition analysis across the 100 runs. This is load-bearing for RQ2.
  2. [§3.3 (Stochastic Dominance Tests)] The FSD/SSD procedure sets dominance as the null hypothesis and treats p > 0.05 as 'implying' dominance. This reverses the actual evidential meaning of a significance test: failing to reject the null is not positive evidence for the null. Consequently, statements such as 'GPT-4o demonstrated first-order stochastic dominance' and 'SSD test further confirmed this pattern' are not supported by the reported tests. The shift-function results in Table 3 do show significant advantages across quantiles, so RQ1 can likely be retained, but the dominance claims should be reframed as 'no significant violation of dominance' or replaced with a procedure that directly tests dominance in the conventional direction.
  3. [§2.3, Table 4, Table 5 (Augmented Intelligence)] The augmented-intelligence condition uses an arbitrary 10:1 human:AI sampling ratio, and because the augmented crowd is 90% human, its performance is expected to track the human crowd. The reported differences between augmented and human-only crowds are tiny: 0.999 vs. 0.998 on RMET and 0.971 vs. 0.960 on MRMET. At the RMET ceiling, a 0.001 difference carries little practical meaning, and the odds-ratio contrasts in Table 5 appear inconsistent with the marginal probabilities. The claim that augmented intelligence 'surpasses' human crowds therefore needs a sensitivity analysis over mixing ratios (e.g., 50:50, 80:20, 90:10, 95:5) and a demonstration that the effect is not a ceiling or construction artifact. This is essential for RQ3.
  4. [§2.3, Table 4 (Crowd-Level GLMM Independence)] The crowd-level GLMM treats each of the resampled group iterations as an independent observation. For GPT-4o, the entire pool is only 100 response vectors, and groups of size 90 drawn with replacement will share most of the same runs across iterations. The random intercept for iteration does not account for the fact that the same underlying response vectors contribute to thousands of supposedly independent rows. This can produce overconfident standard errors and artificially tight confidence intervals in Table 5, and it may directly influence the estimated AI-crowd slope (Table 4: GPT-4o × Log₁₀ Group Size = -3.73 on RMET). The authors should address this by modeling the finite-pool structure, using cluster-robust standard errors at the run level, or resampling from the raw response distribution in a way that respects the actual replication mechanism.
minor comments (5)
  1. [Throughout] The dataset name is inconsistently abbreviated as 'MRET' in several places (e.g., Table 3 note, Figure 1 notes, Figure 2 note); it should be MRMET.
  2. [Table 4] The model information reports 'Observations' of 316,800 for RMET and 325,600 for MRMET, which do not match 6,600 iterations × 36/37 items (237,600 and 244,200). Also, 'Number of Participants 6,600' in a group-iteration model is confusing; this column appears to refer to iterations, not participants. Please clarify the unit of analysis and correct the counts.
  3. [§2.3] The phrase 'with replacement across groups' is ambiguous: it could mean groups are sampled independently (with replacement of group members between groups) or that membership within a group is sampled with replacement. Clarify the exact resampling protocol, especially for the GPT-4o condition where pool size is 100.
  4. [§3.3/Table 5] The reported odds ratios in Table 5 (e.g., 26.015 for RMET Human vs. GPT-4o) are not derivable from the marginal probabilities shown in the same table. Explain how these contrasts were computed and, if they are based on the logit model at a specific group size, state that conditioning.
  5. [§2.2/§5] The data and code availability statement says 'Code and data are available upon request.' Given the paper's reproducibility goals, consider depositing the data and analysis scripts in a public repository.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical measurements with no fitted-parameter-as-prediction or self-citation chain.

full rationale

The paper's central claims—GPT-4o outperforms average humans on RMET/MRMET, human plurality-voting crowds outperform GPT-4o plurality-voting crowds, and a 90:10 human-AI augmented crowd achieves the highest accuracy—are empirical results obtained by applying well-defined aggregation procedures to collected data. No step in the derivation equates an input with an output by definition. The augmented-intelligence condition is a deliberate construction (90% human votes, 10% AI votes), but its measured accuracy is not logically forced by that construction; it is a genuine empirical quantity that could have been lower, equal, or higher than the human-only crowd. The AI-crowd plateau is explained by an asserted item-level consistency of GPT-4o responses, but this assertion is a mechanistic interpretation, not a circular reduction: the paper does not define AI crowd accuracy in terms of that consistency, nor does it fit a parameter and then relabel the fit as a prediction. There are no load-bearing self-citations: the paper's methodological choices (RMET/MRMET, plurality voting, GLMMs) are grounded in external prior work, not in the authors' own previous results. Concerns about pseudoreplication in the crowd-level GLMMs and the lack of item-level response-diversity evidence are statistical-validity or evidence-quality issues, not instances of circularity under the specified taxonomy. The derivation chain is therefore self-contained with respect to the inputs used.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger is light on free parameters because the paper is empirical rather than derived: no constants are fitted. The main hand-selected choice is the 10:1 human:AI mixing ratio in augmented crowds, which shapes the RQ3 result and has no sensitivity analysis. The load-bearing assumptions are statistical and operational: the independence of resampled crowd iterations (likely violated given reuse of the same 100 AI runs and 17,680 human responders), the treatment of 100 near-deterministic runs as an AI crowd distribution, and the reversed-null dominance tests. No new entities are invented.

free parameters (3)
  • Human:AI mixing ratio in augmented crowds = 10:1 (90% human, 10% AI)
    Set by the authors in section 2.3 with no sensitivity analysis; the RQ3 conclusion that augmented crowds outperform human-only crowds depends on this ratio, and other ratios would produce different curves.
  • Crowd group-size range = 5 to 90 members
    Chosen ad hoc; the GLMM log-size slopes (3.99 human vs 0.25 AI on RMET) are estimated over this range and extrapolated in Figure 3.
  • Quantile window for shift-function claims = 0.03 to 0.97, excluding 0.01 and 0.99
    The claim that GPT-4o beats humans at every level is bounded by this window because accuracy saturates at the extremes.
assumptions (4)
  • domain assumption Human responses in the Kim et al. (2024) datasets are independent and scored as in the original validation.
    The analysis inherits 17,680/9,295 human responses without re-validation; any scoring or sampling error upstream propagates into every comparison.
  • domain assumption 100 GPT-4o API runs at temperature 1 and top-p 1 are treated as independent samples of an AI individual for crowd formation.
    The paper itself shows these runs are near-deterministic (section 4), so the AI-crowd comparison measures determinism as much as capability.
  • domain assumption Resampled groups formed with replacement are statistically independent observations for the GLMM in section 2.3.
    Groups are resampled from the same pools (17,680 humans, 100 AI runs) across 6,600 iterations, so the same responses appear in many independent group-level rows; the model does not account for this dependence.
  • ad hoc to paper Failing to reject the null of dominance (p > 0.05) is evidence for stochastic dominance.
    The FSD/SSD tests in section 3.3 reverse the conventional null; absence of evidence against dominance is treated as positive evidence for it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Silicon Minds versus Human Hearts: The Wisdom of Crowds Beats the Wisdom of AI in Emotion Recognition." pith.science (2026). https://pith.science/paper/PZSXPLNC

@misc{pith2026250808830,
  author       = {Pith},
  title        = {Pith review of: Silicon Minds versus Human Hearts: The Wisdom of Crowds Beats the Wisdom of AI in Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZSXPLNC}},
  note         = {Machine review of arXiv:2508.08830}
}
read the original abstract

The ability to discern subtle emotional cues is fundamental to human social intelligence. As artificial intelligence (AI) becomes increasingly common, AI's ability to recognize and respond to human emotions is crucial for effective human-AI interactions. In particular, whether such systems can match or surpass human experts remains to be seen. However, the emotional intelligence of AI, particularly multimodal large language models (MLLMs), remains largely unexplored. This study evaluates the emotion recognition abilities of MLLMs using the Reading the Mind in the Eyes Test (RMET) and its multiracial counterpart (MRMET), and compares their performance against human participants. Results show that, on average, MLLMs outperform humans in accurately identifying emotions across both tests. This trend persists even when comparing performance across low, medium, and expert-level performing groups. Yet when we aggregate independent human decisions to simulate collective intelligence, human groups significantly surpass the performance of aggregated MLLM predictions, highlighting the wisdom of the crowd. Moreover, a collaborative approach (augmented intelligence) that combines human and MLLM predictions achieves greater accuracy than either humans or MLLMs alone. These results suggest that while MLLMs exhibit strong emotion recognition at the individual level, the collective intelligence of humans and the synergistic potential of human-AI collaboration offer the most promising path toward effective emotional AI. We discuss the implications of these findings for the development of emotionally intelligent AI systems and future research directions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 8 canonical work pages

  1. [1]

    wisdom of crowds

    Introduction Emotion recognition, which refers to the ability to discern and interpret others' emotional states, influences how individuals perceive, think, and navigate their social environments (Salovey and Mayer, 1990). People adept at recognizing emotions forge deep connections with others, resolve conflicts, and sustain social cohesion (Lopes et al.,...

    work page 1990

  2. [2]

    Please pick the best possible answer that describes the emotion or mental state the person is experiencing from the following options

    Methods 2.1. Participants and Procedures In this study, we used publicly available data from Kim et al. (2024), who collected 17,680 and 9,295 responses for the RMET and MRMET, respectively, and described original demographic information, recruitment methods, and consent information. For our study, we ran GPT -4o in February 2025 for 50 trials per conditi...

    work page 2024

  3. [3]

    wisdom of crowds

    Results 3.1. AI Outperformed Average Human Performance on Emotion Recognition Tasks To address our first research question about performance across ability levels, we begin by examining overall performance differences between humans and GPT-4o. We evaluated human and GPT - 4o performance differences using RMET and MRMET tasks. Descriptive st atistics and ...

    work page

  4. [4]

    Discussion Our findings addressing the three research questions yield critical insights for understanding AI emotion recognition capabilities and limitations. First, GPT-4o exceeded human performance at all levels on RMET and MRMET, surpassing every performance group from the 3rd to 97th percentiles with first - and second-order stochastic dominance, conf...

    work page 2024

  5. [5]

    Does the Whole Exceed Its Parts? The Effect of AI Explanations on Complementary Team Performance

    References Bansal, Gagan, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. “Does the Whole Exceed Its Parts? The Effect of AI Explanations on Complementary Team Performance.” Pp. 1–16 in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Yokohama Japan: ACM. Baron-Cohe...

    work page 2021

  6. [32]

    How Do We Evaluate Facial Emotion Recognition?

    doi:10.1038/s41562-017-0273-4. Paiva-Silva, Ana Idalina de, Marta Kerr Pontes, Juliana Silva Rocha Aguiar, and Wânia Cristina de Souza. 2016. “How Do We Evaluate Facial Emotion Recognition?” Psychology & Neuroscience 9(2):153–75. doi:10.1037/pne0000047. Peeperkorn, Max, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. “Is Temperature the Creativity P...

    work page doi:10.1038/s41562-017-0273-4 2016

  7. [95]

    GPT-4o Reads the Mind in the Eyes

    doi:10.1038/s41562-024-01882-z. Strachan, James W. A., Oriana Pansardi, Eugenio Scaliti, Marco Celotto, Krati Saxena, Chunzhi Yi, Fabio Manzi, Alessandro Rufo, Guido Manzi, Michael S. A. Graziano, Stefano Panzeri, and Cristina Becchio. 2024. “GPT-4o Reads the Mind in the Eyes.” Surowiecki, James. 2005. The Wisdom of Crowds. Vintage. Tamkin, Alex, Miles Br...

    arXiv 2024

  8. [2004]

    Emotional Intelligence and Social Interaction

    “Emotional Intelligence and Social Interaction.” Personality and Social Psychology Bulletin 30(8):1018–34. Meng, Qingqi, Tung-Ju Wu, Wenyan Duan, and Shijia Li. 2025. “Effects of Employee–Artificial Intelligence (AI) Collaboration on Counterproductive Work Behaviors (CWBs): Leader Emotional Support as a Moderator.” Behavioral Sciences 15(5):696. doi:10.33...

    doi:10.3390/bs15050696 2025

Show all 10 references
  1. [2020]

    Language Models Are Few-Shot Learners

    “Language Models Are Few-Shot Learners.” Pp. 1877–1901 in Advances in Neural Information Processing Systems. Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin. Curran Associates, Inc. Centola, Damon. 2018. How Behavior Spreads: The Science of C...

    1901

  2. [2024]

    To account for the uneven sample sizes, we used mixed-effects models that adjust through partial pooling, allowing flexible group-size handling (Pinheiro and Bates, 2000)

    shows statistical performance gains for LLMs diminish sharply after a few runs, making large samples unnecessary. To account for the uneven sample sizes, we used mixed-effects models that adjust through partial pooling, allowing flexible group-size handling (Pinheiro and Bates...

    2000

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.