Pith. sign in

REVIEW 2 major objections 5 minor 41 references

Common diversity measures for LLM ensembles mostly re-express capability: strict diversity is nearly one minus mean accuracy, and after control only more shared error robustly predicts lower majority-vote gain.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:24 UTC pith:VD2JDSUV

load-bearing objection A careful, honest audit showing diversity metrics mostly re-express capability in LLM pools; the core collinearity is real but the non-neutral parse slice is a legitimate caveat. the 2 major comments →

arxiv 2607.20768 v1 pith:VD2JDSUV submitted 2026-07-22 cs.CL cs.AIcs.LG

Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles

Donghwan Kim This is my paper
classification cs.CL cs.AIcs.LG
keywords majority votingLLM ensemblesdiversity measurescapability confoundingco-failureMMLU-Proensemble selectioncontingency table
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Everyone knows that diverse models should vote better than any single model, so diversity scores are used to pick LLM ensembles. This paper argues that, on modern LLM pools, the most commonly used diversity scores are mostly just measuring average member skill: strict diversity, for instance, is a nearly perfect mirror of one minus mean accuracy, leaving almost no independent variation once capability is controlled. After that control, the only signal that reliably predicts whether majority voting beats the strongest member is a modest negative one: pairs of models that share more errors produce lower gains. In other words, raw diversity–gain correlations are misleading, and the practical takeaway is to evaluate ensembles against the best member, control for capability, and treat contingency-table diversity measures as algebraically coupled rather than independent quantities.

Core claim

The paper audits five diversity-related statistics — strict diversity (complement of joint correctness), disagreement, double-fault (co-failure), pairwise Jaccard error overlap, and focal diversity — as predictors of majority-vote gain over the best member across all 31,900 size-2–4 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA). It establishes that the three contingency-table statistics are linearly dependent: strict = disagreement + double-fault and 1 − mean accuracy = double-fault + ½ disagreement, so raw-space linear control for capability forces a one-dimensional residual and joint regressions are rank-deficient. Empirically, strict diversity is nearly collinear with one minus mean a

What carries the argument

The central object is the 2×2 contingency table of pairwise joint correctness (a: both correct, b/c: split errors, d: both wrong). Two exact algebraic identities do the heavy lifting: strict diversity = disagreement + double-fault, and 1 − mean accuracy = double-fault + ½ disagreement. The first makes three audited statistics linearly dependent, so raw-space regressions treating them as independent predictors are rank-deficient; the second forces any linear residualization that controls mean accuracy to produce a perfectly collinear residual axis (Pearson r = −1, slope −1/2). The empirical complement is that on modern LLM pools, strict diversity is nearly collinear with 1 − mean accuracy (ρ

Load-bearing premise

The headline numbers come from the 356-item common-parse slice created by keeping only items every model could answer; the 144 dropped items are harder (mean accuracy 0.659 vs. 0.791) and more disagreed-over (0.316 vs. 0.178), and if the regime where diversity information lives is exactly this filtered-out region, the measured capability entanglement and the residual co-failure direction would be slice-induced.

What would settle it

Recompute the strict-diversity vs. 1−mean-accuracy Spearman correlation and the capability-controlled double-fault partial correlation on the 144 dropped hard items alone, or on a harder benchmark with per-subset denominators. If ρ drops well below ~0.99 on those items, or if the controlled double-fault association becomes positive or near-zero in that regime, the paper's central claim of capability entanglement with no separable diversity signal would be falsified for exactly the items where diversity would matter most.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Raw diversity–gain correlations should not be read as evidence that diversity hurts or helps: most associations flip or vanish once member capability is controlled.
  • Because strict, disagreement, and double-fault are algebraically coupled and mostly re-express mean accuracy, using them as three independent predictors in a regression is rank-deficient and meaningless.
  • Majority voting converts latent complementarity into a win only rarely: oracle gain is positive in 100% of subsets, but the vote beats the best member in just 9.98% of size-3 subsets (18.71% when the best member is chosen on a held-out split).
  • The one directionally robust residual signal is pairwise co-failure: more shared error predicts lower majority-vote gain, so ensemble-selection heuristics should focus on reducing pairwise error overlap rather than maximizing generic diversity.
  • Capability controls, including nonlinear ones, should be standard in any future claim that a diversity measure drives ensemble gain.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the near-collinearity of joint-correctness with mean accuracy holds across mainstream LLM pools, then any diversity measure defined over joint correctness is redundant with average skill for selection; the useful signal is not 'diversity' but the structure of shared errors, which only shows up in the double-fault cell.
  • The paper's evidence suggests a concrete selection rule to test: among subsets of comparable mean and best accuracy, pick the one with the smallest pairwise double-fault residual; the paper's own held-out AUC of 0.597 shows this is a weak but non-random predictor of rare positive gains, and a direct interventional study could quantify its lift.
  • The finding that the co-failure signal weakens on hard items implies that filtering by parseability may be removing exactly the items where ensembles could differentiate models; a deliberate sampling design that oversamples hard, high-disagreement items would test whether the entanglement is intrinsic or slice-induced.
  • The algebraic non-separability result generalizes beyond LLMs to any ensemble of classifiers where the three contingency-table statistics are computed, so the audit method transfers to classical ensemble learning and other prediction settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper asks whether five diversity-related statistics (strict diversity, disagreement, double-fault, pairwise Jaccard error-set similarity, and focal diversity) predict the majority-vote gain over the best member in LLM ensembles, once member capability is controlled. The audit enumerates all 31,900 size-2--4 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) and applies six linear control specifications plus nonlinear, matched, model-resampling, and leave-one-model-out robustness checks. Three headline findings are reported: (i) oracle complementarity is universal but realized majority-vote gain over the best member is rare (9.98% at size 3 under in-sample best selection); (ii) strict diversity is nearly collinear with one minus mean accuracy (Spearman rho = +0.991/+0.988); and (iii) after capability control, the only directionally stable contingency-table signal is a modest residual pairwise co-failure association, with more shared error associated with lower gain. The paper is careful to separate algebraic identities from empirical claims, and it repeatedly stresses that the findings are associational and slice-conditional.

Significance. If the results hold, the paper makes a useful measurement-level contribution: several diversity metrics commonly used to select LLM ensembles largely re-express member capability in current model pools, and the linear coupling among strict diversity, disagreement, and double-fault makes raw-space joint regressions rank-deficient by construction. The paper's strengths include exhaustive subset enumeration, explicit separation of algebraic identities from empirical regularities, a wide battery of control and robustness specifications, model-level resampling rather than inflated subset-level p-values, cross-benchmark reproduction on TruthfulQA, and a planned release of scripts and derived correctness matrices sufficient for independent reproduction. The authors are also unusually candid about the non-neutrality of their item filter and other limitations. The main risks are data-conditionality issues concerning which items and which prompt versions enter the correctness matrix.

major comments (2)
  1. [Section 4 / Table 1 / Section 5.6] All headline numbers, including the keystone collinearity strict-diversity vs. 1-mean-accuracy (rho = +0.991/+0.988) and the residual double-fault association (-0.432), are computed on the 356-item common-parse slice. Table 1 shows the 144 dropped items are substantially harder (mean accuracy 0.659 vs. 0.791) and more disagreed (0.316 vs. 0.178), so the filter is not neutral. The checks in Section 5.6 (per-subset denominators, 451-item slice) still select on parsed items and cannot fully rule out slice-induced inflation; Limitation 1 concedes the issue is 'mitigated, not eliminated.' I request a full-500 analysis scoring unparsed responses as incorrect, consistent with the full-500 accuracy definitions in Appendix A.1, or a formal argument why that specification would be invalid. This is load-bearing because both the entanglement diagnosis and the residual co-failure claim are reported o
  2. [Appendix A.2 / Section 8, Limitation 6] The retry protocol re-queried every previously unparsed response with progressively simplified prompts, with Retry 2 and Retry 3 dropping the chain-of-thought instruction. The final correctness matrix therefore mixes initial-prompt responses with simplified-prompt responses, and the pre-retry intersection parsed by all 30 models is only 18 items. A per-subset initial-response-only analysis would not require all 30 models to have parsed an item, so the 18-item figure does not by itself justify omitting such an analysis. Please report the number and fraction of retry-derived responses in the common slice and add a sensitivity analysis using only initial responses on per-subset denominators, or a unified re-prompting of a subsample. Without this, the internal comparability of model predictions underlying every result is uncertain.
minor comments (5)
  1. [Appendix A.1] The MMLU-Pro roster uses anthropic/claude-haiku-4.5 while the TruthfulQA roster uses anthropic/claude-haiku-4-5. Please clarify whether these are the same underlying model; if not, the TruthfulQA 'reproduction' uses a slightly different roster beyond the exclusion of qwen3.6-plus, and this should be stated explicitly.
  2. [Table 3 footnote] The dagger footnote 'Linear-control positive residuals in the full pool only' is cryptic. The text explains that the positive strict/disagreement residuals are roster-dependent, but the footnote should say this directly, since readers may otherwise interpret the +0.339/+0.292 values as robust effects.
  3. [Section 3.1] The focal diversity definition would benefit from one sentence of intuition: rho_i measures, for items on which member i fails, how rarely other members also fail, normalized so that fully disjoint failures give high diversity. Currently the formula is given without a plain-language interpretation.
  4. [Figure 4] The figure reports a descriptive in-sample R^2 from an OLS projection onto ranked best and mean accuracy. The text states this, but the axis label 'variance accounted for' may be misread as predictive or held-out. Consider labeling it 'descriptive in-sample R^2' on the figure itself.
  5. [Appendix C] The 'attenuation-corrected (approx.)' value of -0.53 is based on the classical attenuation formula applied to a partial correlation, which the paper notes treats the controls as measured without error. This is a useful sanity check, but the caveat should appear next to the table entry as well as in the text.

Circularity Check

0 steps flagged

No significant circularity: algebraic identities are explicitly labeled algebraic and empirical claims are tested, not fitted.

full rationale

The paper's derivation chain is not circular. The two identities in Section 3.1 (strict = disagreement + double-fault; 1 − Acc = double-fault + 1/2 disagreement) are presented as exact algebra, and Section 5.4 explicitly says: 'This raw-space one-dimensionality is algebraically inevitable; it is not an empirical discovery.' The near-collinearity of strict diversity with 1 − mean accuracy is treated as an empirical registry property of the pool, not derived from the identities alone, and it is probed with per-subset denominators, a less-filtered 451-item slice, threshold changes, and a second benchmark. The residual pairwise co-failure association is a measured partial-Spearman quantity after rank-space residualization, not a fitted parameter renamed as a prediction; the paper repeatedly stresses its direction is robust but its magnitude is configuration-dependent. The only self-citation (Kim, 2026) is an illustrative aside about closed-loop table recognition and is not load-bearing for the audit's claims. The common-parse slice is a real data-selection threat, acknowledged as 'mitigated, not eliminated' in Section 8, but selection bias is a validity risk rather than definitional circularity. No load-bearing step reduces to its own input by construction.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The central claims rest on accuracy-based capability controls, rank-space residualization, a non-neutral common-parse slice, and a specific OpenRouter roster. No fitted model parameters or new theoretical entities are introduced; the only hand-chosen input affecting the roster is the accuracy band, with sensitivity analysis.

free parameters (1)
  • Accuracy inclusion band [0.40, 0.92] = 0.40, 0.92
    Pragmatic model-inclusion filter, not preregistered or literature-derived (Section 4). Non-binding on MMLU-Pro; on TruthfulQA it excludes qwen3.6-plus (full-500 accuracy 0.922). A post-hoc no-upper-cutoff sensitivity check is reported; still a hand-chosen threshold that affects roster composition.
axioms (5)
  • domain assumption Member accuracy (best/mean/min/median/std/full vector) adequately operationalizes 'capability' for confounding control.
    If capability has components not captured by accuracy on the same benchmark, residual diversity associations could be confounded rather than diversity-specific. The paper controls only on accuracy-based quantities.
  • standard math Rank-space linear residualization (partial Spearman) is a valid control procedure for heavily overlapping subset data.
    Used throughout; the paper avoids subset-level p-values because subsets reuse 30 models, and supports inference with model-level resampling intervals and leave-one-model-out.
  • domain assumption The common parsed item slice (356/338 items) is representative enough for estimating the associations.
    Section 4/Table 1: the filter is not neutral—dropped items are harder and more disagreed—and the authors say the issue is mitigated, not eliminated, with per-subset denominators and a less-filtered 451-item slice.
  • domain assumption The 30/29-model rosters via OpenRouter default routing represent the population of modern LLMs relevant to ensemble selection.
    Roster composition is provider-dependent and time-specific; the paper itself shows near-zero co-failure association in a restricted Chinese-origin pool, so magnitudes are pool-sensitive.
  • domain assumption Majority vote under the correct-count rule with gain over the best member is a meaningful baseline for auditing diversity.
    This defines the dependent variable and deliberately makes the bar stringent; alternatives (held-out best, plurality rule) are reported and change rates but not the qualitative conclusions.

pith-pipeline@v1.3.0-alltime-deepseek · 18977 in / 15228 out tokens · 138084 ms · 2026-08-01T09:24:33.906898+00:00 · methodology

0 comments
read the original abstract

Majority voting over LLMs is widely assumed to benefit from diversity, and diversity measures are used to choose which models to combine. We ask whether five such measures track diversity or mainly re-express capability, auditing them as predictors of majority-vote gain over the best member across 31,900 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) under explicit capability controls. Three findings emerge. First, latent complementarity is ubiquitous: oracle gain is positive in 100% of subsets, yet simple voting beats the strongest member in only 9.98% of all canonical size-3 subsets (18.71% with held-out best selection); the pooled size-2-4 rate is 1.27%, partly reflecting deterministic even-size voting behavior. Second, a joint-correctness proxy (strict diversity) is nearly collinear with one minus mean accuracy (size-3 Spearman rho = +0.991 / +0.988); raw diversity-gain associations are strongly capability-entangled and, with one exception, unstable under control. Third, three linear contingency-table statistics are algebraically non-separable; after capability control, the empirically stable remainder is a modest residual pairwise co-failure association in which more shared error corresponds to lower gain. This direction is robust, but its magnitude is configuration-dependent. Joint rawspace linear regressions treating strict diversity, disagreement, and double-fault as independent predictors are rank-deficient by construction.

Figures

Figures reproduced from arXiv: 2607.20768 by Donghwan Kim.

Figure 1
Figure 1. Figure 1: Capability-controlled audit pipeline. Top: benchmark-specific routes, fixed filters, and common-parse [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Oracle opportunity versus realized majority [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Strict diversity against one minus mean member accuracy, size 3, on MMLU-Pro (left) and TruthfulQA [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Capability control changes apparent diversity–gain associations (size 3; values from Ta￾ble 3): raw vs. partial Spearman controlling best and best+mean. Only double-fault stays negative through￾out. the association is strong on easy/medium items (≈−0.75 to −0.86) but much weaker on hard items (−0.18 to −0.22), where gain variation is lim￾ited (Appendix Figure C3); the medium band is small (36–42 items) due… view at source ↗
Figure 6
Figure 6. Figure 6: Residual pairwise co-failure association (size [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 1 canonical work pages

  1. [1]

    Beyond Majority Voting:

    Ai, Rui and Pan, Yuqi and Simchi-Levi, David and Tambe, Milind and Xu, Haifeng , booktitle =. Beyond Majority Voting:. 2026 , month = jul, eprint =

    2026

  2. [3]

    2026 , month = jul, url =

    Baek, David and Estornell, Andrew and Zhang, Yichi and Taufiq, Muhammad Faaiz and Ton, Jean-Francois and Mei, Jie and Wang, Tao , booktitle =. 2026 , month = jul, url =

    2026

  3. [5]

    2026 , month = jun, eprint =

    When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models , author =. 2026 , month = jun, eprint =

    2026

  4. [6]

    ICML 2024 Workshop on In-Context Learning , year =

    Universal Self-Consistency for Large Language Models , author =. ICML 2024 Workshop on In-Context Learning , year =

    2024

  5. [10]

    2023 , address =

    Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen , booktitle =. 2023 , address =. doi:10.18653/v1/2023.acl-long.792 , url =. 2306.02561 , archiveprefix =

    Pith/arXiv arXiv 2023

  6. [13]

    Advances in Neural Information Processing Systems 7 , editor =

    Neural Network Ensembles, Cross Validation, and Active Learning , author =. Advances in Neural Information Processing Systems 7 , editor =. 1995 , publisher =

    1995

  7. [18]

    2024 , address =

    Tekin, Selim Furkan and Ilhan, Fatih and Huang, Tiansheng and Hu, Sihao and Liu, Ling , booktitle =. 2024 , address =. doi:10.18653/v1/2024.findings-emnlp.698 , url =. 2410.03953 , archiveprefix =

    Pith/arXiv arXiv 2024

  8. [23]

    Journal of Machine Learning Research , year =

    A Unified Theory of Diversity in Ensemble Learning , author =. Journal of Machine Learning Research , year =

  9. [24]

    Journal of Machine Learning Research , year =

    Ensemble Pruning Via Semi-definite Programming , author =. Journal of Machine Learning Research , year =

  10. [28]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Improving Factuality and Reasoning in Language Models through Multiagent Debate , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , volume =

    2024

  11. [32]

    Rui Ai, Yuqi Pan, David Simchi-Levi, Milind Tambe, and Haifeng Xu. 2026. https://arxiv.org/abs/2510.01499 Beyond majority voting: LLM aggregation by leveraging higher-order information . In Forty-Third International Conference on Machine Learning

    Pith/arXiv arXiv 2026

  12. [33]

    Junade Ali. 2026. https://arxiv.org/abs/2607.17384 Quantifying diversity of thought: A predictive law of weighted LLM ensemble lift . Preprint, arXiv:2607.17384

    Pith/arXiv arXiv 2026

  13. [34]

    David Baek, Andrew Estornell, Yichi Zhang, Muhammad Faaiz Taufiq, Jean-Francois Ton, Jie Mei, and Tao Wang. 2026. https://icml.cc/virtual/2026/poster/63783 D-FUSEr : Diverse failure, unified success via error-distribution shaping in LLM reasoning . In Forty-Third International Conference on Machine Learning. OpenReview: To2O1ed5cV

    2026

  14. [35]

    Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. 2005. https://doi.org/10.1016/j.inffus.2004.04.004 Diversity creation methods: a survey and categorisation . Information Fusion, 6(1):5--20

    doi:10.1016/j.inffus.2004.04.004 2005

  15. [36]

    Josef Chen. 2026. https://arxiv.org/abs/2606.27288 When does combining language models help? a co-failure ceiling on routing, voting, and mixture-of-agents across 67 frontier models . Preprint, arXiv:2606.27288

    Pith/arXiv arXiv 2026

  16. [37]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2024 a . https://arxiv.org/abs/2305.05176 FrugalGPT : How to use large language models while reducing cost and improving performance . Transactions on Machine Learning Research

    Pith/arXiv arXiv 2024

  17. [38]

    Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2024 b . https://openreview.net/forum?id=LjsjHF7nAN Universal self-consistency for large language models . In ICML 2024 Workshop on In-Context Learning. Earlier version titled ``Universal Self-Consistency for Large Langua...

    Pith/arXiv arXiv 2024

  18. [39]

    Zhijun Chen, Xiaodong Lu, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Xiao Huang, Yikun Ban, Hailong Sun, and Philip S. Yu. 2025. https://arxiv.org/abs/2502.18036 Harnessing multiple large language models: A survey on LLM ensemble . Preprint, arXiv:2502.18036. Version 6, updated 22 April 2026

    Pith/arXiv arXiv 2025

  19. [40]

    Seffi Cohen, Nurit Cohen Inger, Niv Goldshlager, Bracha Shapira, and Lior Rokach. 2026. https://doi.org/10.18653/v1/2026.findings-eacl.282 DFPE : A diverse fingerprint ensemble for enhancing LLM performance . In Findings of the Association for Computational Linguistics: EACL 2026 , pages 5326--5336, Rabat, Morocco. Association for Computational Linguistics

    work page doi:10.18653/v1/2026.findings-eacl.282 2026

  20. [41]

    Dietterich

    Thomas G. Dietterich. 2000. https://doi.org/10.1007/3-540-45014-9_1 Ensemble methods in machine learning . In Multiple Classifier Systems, volume 1857 of Lecture Notes in Computer Science, pages 1--15, Berlin, Heidelberg. Springer

    doi:10.1007/3-540-45014-9_1 2000

  21. [42]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. https://proceedings.mlr.press/v235/du24e.html Improving factuality and reasoning in language models through multiagent debate . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11733--11...

    2024

  22. [43]

    Shashwat Goel, Joschka Str \"u ber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. 2025. https://arxiv.org/abs/2502.04313 Great models think alike and this undermines AI oversight . In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceeding...

    Pith/arXiv arXiv 2025

  23. [44]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. https://doi.org/10.18653/v1/2023.acl-long.792 LLM-Blender : Ensembling large language models with pairwise ranking and generative fusion . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165--14178, Toronto, Canada. Association f...

    doi:10.18653/v1/2023.acl-long.792 2023

  24. [45]

    Donghwan Kim. 2026. https://arxiv.org/abs/2607.13347 LLM -as-a-judge scores are unreliable optimization signals in closed-loop table recognition . Preprint, arXiv:2607.13347

    Pith/arXiv arXiv 2026

  25. [46]

    Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. 2025. https://arxiv.org/abs/2506.07962 Correlated errors in large language models . In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 30038--30066. PMLR

    Pith/arXiv arXiv 2025

  26. [47]

    Anders Krogh and Jesper Vedelsby. 1995. https://proceedings.neurips.cc/paper/1994/hash/b8c37e33defde51cf91e1e03e51657da-Abstract.html Neural network ensembles, cross validation, and active learning . In Advances in Neural Information Processing Systems 7, pages 231--238. MIT Press. Proceedings of the 1994 conference

    1995

  27. [48]

    Kuncheva and Christopher J

    Ludmila I. Kuncheva and Christopher J. Whitaker. 2003. https://doi.org/10.1023/A:1022859003006 Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy . Machine Learning, 51(2):181--207

    doi:10.1023/a:1022859003006 2003

  28. [49]

    Baptiste Lefort, Eric Benhamou, Jean-Jacques Ohana, B \'e atrice Guez, David Saltiel, and Thomas Jacquot. 2024. https://doi.org/10.48550/arXiv.2409.00094 Examining independence in ensemble sentiment analysis: A study on the limits of large language models using the condorcet jury theorem . Preprint, arXiv:2409.00094

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.00094 2024

  29. [50]

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. 2024. https://arxiv.org/abs/2402.05120 More agents is all you need . Transactions on Machine Learning Research

    Pith/arXiv arXiv 2024

  30. [51]

    Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. 2026. https://arxiv.org/abs/2502.00674 Rethinking mixture-of-agents: Is mixing different large language models beneficial? Transactions on Machine Learning Research

    Pith/arXiv arXiv 2026

  31. [52]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 TruthfulQA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214--3252, Dublin, Ireland. Association for Computational Linguistics

    doi:10.18653/v1/2022.acl-long.229 2022

  32. [53]

    Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.552 Getting MoRE out of mixture of language model reasoning experts . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8234--8249, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.552 2023

  33. [54]

    Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.698 LLM-TOPLA : Efficient LLM ensemble by maximising diversity . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11951--11966, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-emnlp.698 2024

  34. [55]

    Yigit Turkmen, Baturalp Buyukates, and Melih Bastopcu. 2026. https://arxiv.org/abs/2602.08003 Don't always pick the highest-performing model: An information theoretic view of LLM ensemble selection . Preprint, arXiv:2602.08003

    arXiv 2026

  35. [56]

    Vardeman and Max D

    Stephen B. Vardeman and Max D. Morris. 2013. https://doi.org/10.1080/00031305.2013.778788 Majority voting by independent classifiers can increase error rates . The American Statistician, 67(2):94--96

    Pith/arXiv arXiv 2013

  36. [57]

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2025. https://arxiv.org/abs/2406.04692 Mixture-of-agents enhances large language model capabilities . In International Conference on Learning Representations. Spotlight

    Pith/arXiv arXiv 2025

  37. [58]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . In International Conference on Learning Representations

    Pith/arXiv arXiv 2023

  38. [59]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://arxiv.org/abs/2406.01574 MMLU-Pro : A more robust and challenging multi-task language understanding benchmark . In Advances in Neura...

    Pith/arXiv arXiv 2024

  39. [60]

    Webb, Henry W

    Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luj \'a n, and Gavin Brown. 2023. https://jmlr.org/papers/v24/23-0041.html A unified theory of diversity in ensemble learning . Journal of Machine Learning Research, 24(359):1--49

    2023

  40. [61]

    Nick Street

    Yi Zhang, Samuel Burer, and W. Nick Street. 2006. https://jmlr.org/papers/v7/zhang06a.html Ensemble pruning via semi-definite programming . Journal of Machine Learning Research, 7(48):1315--1338

    2006

  41. [62]

    Yichi Zhang, Kevin Lu, Yuang Zhang, Jie Gao, Lirong Xia, and Fang-Yi Yu. 2026. https://arxiv.org/abs/2605.24048 Mixture of complementary agents for robust LLM ensemble . Preprint, arXiv:2605.24048. Associated ICLR 2026 submission withdrawn

    Pith/arXiv arXiv 2026