pith. sign in

arxiv: 2605.27394 · v1 · pith:24ENA43Ynew · submitted 2026-04-19 · 💻 cs.CY · cs.AI· cs.HC· cs.MA

Human-AI Collaboration for Estimating Scientific Replicability

Tatiana Chakravorti , Robert Fraleigh , Timothy Fritton , Christopher Griffin , Vaibhav Singh , Sai Koneru , C. Lee Giles , David Pennock
show 2 more authors
Anthony Kwasnica Sarah Rajtmajer
This is my paper

Pith reviewed 2026-07-05 17:12 UTC · model glm-5.2

classification 💻 cs.CY cs.AIcs.HCcs.MA
keywords humanhybridparticipantsreplicationscientificwhileagentsapproach
0
0 comments X

The pith

AI and human traders jointly forecast which studies replicate

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a hybrid prediction market where algorithmic agents (trained on 402 prior replication outcomes) and human domain experts trade contracts on whether a given published finding will survive a controlled replication. The market price becomes a live forecast. Across six disciplines, the authors compare hybrid markets to AI-only and human-only markets and find that, in most disciplines, the hybrid configuration achieves lower or comparable mean absolute error. The central claim is that combining machine pattern recognition with human contextual judgment through a shared trading mechanism produces forecasts at least as good as either component alone.

Core claim

The central object is the hybrid prediction market: a 12-hour live trading environment where algorithmic agents (instantiated in feature space at training data points and using a logarithmic market scoring rule) buy and sell replication contracts alongside human PhD-level researchers who have read the paper. The final price of a 'will replicate' contract serves as the probability estimate. The paper reports that hybrid markets achieved the lowest MAE in sociology and political science, were competitive in economics, and underperformed AI-only markets in marketing and education. Survey evidence shows human participants traded primarily on epistemic beliefs about replicability rather than pure

What carries the argument

hybrid prediction market

Load-bearing premise

The claim that hybrid markets 'consistently match or outperform' AI-only markets rests on comparisons using five test studies per discipline, with no confidence intervals or significance tests reported, making it impossible to distinguish genuine superiority from sampling noise.

What would settle it

If hybrid markets show no systematic MAE improvement over AI-only markets when tested on a larger sample of replication studies per discipline, the consistency claim fails.

Figures

Figures reproduced from arXiv: 2605.27394 by Anthony Kwasnica, Christopher Griffin, C. Lee Giles, David Pennock, Robert Fraleigh, Sai Koneru, Sarah Rajtmajer, Tatiana Chakravorti, Timothy Fritton, Vaibhav Singh.

Figure 1
Figure 1. Figure 1: Forty-one features were extracted from full text and corresponding metadata for each research claim in the training and test datasets. (see [62]) . 3. Data Algorithmic agents were trained on the outcomes of 402 replication studies of published findings in the social and behavioral sciences. In particular, the training dataset included outcomes of 313 studies assembled from the Reproducibility Project Psych… view at source ↗
Figure 2
Figure 2. Figure 2: for notional representation). Agents are sensitive to asset price, which causes their bid behavior to evolve in time [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Schematic representation of hybrid markets for replication prediction. Step 1: Features are extracted from full text of a paper of interest. Step 2: Extracted features are passed to algorithmic traders; the full text PDF is provided to human traders. Step 3: Algorithmic and human traders buy and sell contracts representing ”will replicate” or ”will not replicate” outcomes of a replication study associated … view at source ↗
Figure 4
Figure 4. Figure 4: Interactive web application. Human participants are provided with information about the replica￾tion study and full text of the associated paper. We provide them with initial cash to invest. They may buy and sell contracts representing ‘will replicate’ and ‘will not replicate’ outcomes of the study via the app. education. Hybrid markets achieved the lowest prediction error in sociology and politi￾cal scien… view at source ↗
read the original abstract

Determining whether published scientific findings can successfully be replicated is a long-standing challenge in the empirical sciences. Existing approaches for replicability assessment typically rely either on human judgment, i.e., creative assembly of human experts, or on machine learning models trained on paper content metadata. While both approaches have demonstrated value, each also has important limitations. Human forecasts can be influenced by cognitive biases and narrow exposure to the research literature, while automated assessments often struggle to capture contextual cues and subtle signals of credibility. In this paper, we examine a hybrid approach. Specifically, we introduce a hybrid prediction market in which algorithmic agents trade alongside human participants to jointly estimate the likelihood that a published scientific finding will be corroborated via the outcome of a controlled replication study. Agents are trained on outcomes from hundreds of prior replication studies while human participants contribute domain knowledge through real-time trading. We evaluate this hybrid approach through multiple live experiments involving participants from different academic disciplines and compare its performance to artificial-only and human-only baselines. Our results show that, except for a few cases, hybrid markets match or outperform artificial prediction markets, producing more accurate and reliable replication forecasts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 7 minor

Summary. This paper introduces a hybrid human-AI prediction market for forecasting scientific replication outcomes. Algorithmic agents, trained on 402 prior replication studies using a geometric artificial prediction market framework from prior work, trade alongside human domain experts in a live market platform. The system is evaluated on 30 held-out replication studies across six disciplines (5 per discipline), comparing hybrid markets against artificial-only and (for 3 disciplines) human-only baselines using mean absolute error (MAE). The authors find that hybrid markets match or outperform artificial-only markets in most but not all domains, and present survey evidence about participant trading strategies.

Significance. The paper makes a genuine infrastructure contribution: the hybrid market platform enabling live bot-human trading is novel, and the experimental design — recruiting domain experts to trade on real replication outcomes — is well-motivated. The problem of replicability forecasting is timely and the human-AI complementarity framing is well-grounded in the literature. The feature extraction pipeline and artificial market architecture are drawn from the authors' prior work, which is appropriately cited. The paper reports falsifiable predictions with ground-truth outcomes and provides per-study final prices (Tables 2–3), enabling independent verification. However, the central empirical claim of consistent hybrid improvement is not adequately supported by the evidence as currently presented, as detailed below.

major comments (4)
  1. §5.1, Table 4: The central claim that hybrid markets 'consistently match or outperform' artificial markets is not supported by the evidence at current sample sizes. With n=5 studies per discipline and no confidence intervals, p-values, or any inferential statistics, the MAE differences (e.g., economics 0.411 vs 0.452; marketing 0.490 vs 0.430) cannot be distinguished from noise. The standard error of a mean from bounded outcomes with n=5 is on the order of 0.15–0.20, making differences of 0.04–0.09 statistically indistinguishable from zero. The authors should either (a) report inferential statistics, bootstrap confidence intervals, or permutation tests, or (b) substantially soften the claim from 'consistently match or outperform' to something the data can support, such as 'showed comparable performance in a preliminary evaluation.'
  2. §5.1, Table 4: Hybrid markets lost to AI-only in 2 of 6 domains (marketing: 0.490 vs 0.430; education: 0.488 vs 0.458), which is one-third of tested domains. The abstract's phrase 'except for a few cases' understates this proportion. The framing should be made consistent with the actual results; two out of six is not 'a few' in a way that supports a claim of consistency.
  3. §5.1, Table 4: In 2 of the 3 domains where human-only baselines were run, human-only markets matched or outperformed hybrid markets (economics: 0.414 vs 0.411; psychology: 0.378 vs 0.523). This raises a direct question about whether AI agents add value or introduce noise relative to human-only markets. The paper does not address this finding. A discussion of when and why hybrid markets underperform human-only baselines is needed for the central claim about the value of human-AI collaboration to hold.
  4. §4.2: Several hyperparameters (lambda, liquidity, percent difference, initial agent cash, market duration) were tuned for the hybrid experiments, but the tuned values are not reported, and the sensitivity of results to these choices is not discussed. Given that these parameters directly affect the balance between agent and human influence on market prices, their omission makes the results difficult to interpret and reproduce. The specific values should be reported, and ideally a sensitivity analysis or at least a justification for the chosen configuration should be provided.
minor comments (7)
  1. Table 2: The column 'Final Pred. Human' appears to contain prediction classifications (R/NR) but the header is ambiguous given that the preceding columns are 'Final Price.' Consider relabeling to 'Human Final Prediction' or similar for clarity.
  2. §3, Table 1: The training data is heavily skewed toward psychology (252 of 402) and economics (99), with only 5–8 studies in several other domains. This imbalance likely affects agent performance across domains and should be discussed, particularly given that hybrid markets underperformed in marketing and education, two domains with minimal training representation.
  3. §4.3: The minimum activity rule (three trades) is mentioned but the impact of non-trading participants on market dynamics is not analyzed. How many participants across all sessions failed to meet this threshold, and were their cash allocations still in the market?
  4. §5.2: The survey analysis is purely qualitative. Quantifying the distribution of reported strategies (e.g., percentage of participants citing each strategy) would strengthen this section.
  5. Figure 2: The caption mentions high-dimensional feature space projected down for visualization, but the axes are unlabeled. Please label axes explicitly.
  6. §6: The limitations section mentions small sample size and variable engagement but does not mention the absence of statistical significance testing, which is arguably the most significant limitation.
  7. The paper would benefit from a brief comparison to other replication prediction methods (e.g., text-based ML models cited in §2.1) in terms of MAE, to contextualize the hybrid market's performance against the broader literature, not just against the artificial market baseline.

Simulated Author's Rebuttal

4 responses · 1 unresolved

We thank the referee for a careful and constructive review. The referee correctly identifies the infrastructure contribution and experimental design as strengths, and raises four major comments focused on inferential statistics, framing consistency, the human-only baseline comparison, and hyperparameter reporting. We agree with the substance of all four comments and will revise accordingly. Specifically, we will (1) add bootstrap confidence intervals and permutation tests, (2) soften the central claim from 'consistently match or outperform' to language reflecting a preliminary evaluation, (3) add discussion of cases where human-only markets outperform hybrid markets, and (4) report all tuned hyperparameter values with justification. We cannot fully resolve the fundamental limitation of small sample sizes (n=5 per discipline), but we will be transparent about this constraint and frame conclusions accordingly.

read point-by-point responses

  1. Referee: §5.1, Table 4: The central claim that hybrid markets 'consistently match or outperform' artificial markets is not supported by the evidence at current sample sizes. With n=5 studies per discipline and no confidence intervals, p-values, or any inferential statistics, the MAE differences cannot be distinguished from noise. The authors should either (a) report inferential statistics, bootstrap confidence intervals, or permutation tests, or (b) substantially soften the claim.

    Authors: The referee is correct on both counts. With n=5 studies per discipline, the MAE differences we report (e.g., economics 0.411 vs. 0.452; marketing 0.490 vs. 0.430) are well within the range expected from sampling noise alone. We should not have used the word 'consistently' to describe these results, and we should have reported inferential statistics. We will address this in two ways. First, we will add bootstrap confidence intervals for each MAE estimate and permutation tests for the pairwise comparisons between hybrid and artificial markets. We expect these will confirm that the differences are not statistically significant at conventional thresholds, which is consistent with the referee's calculation that standard errors are on the order of 0.15–0.20. Second, we will revise the central claim throughout the paper — in the abstract, Section 5.1, and the conclusions — from 'consistently match or outperform' to language such as 'showed comparable performance to artificial-only markets in a preliminary evaluation with limited sample sizes.' We agree that the data as presented do not support a claim of consistent improvement. revision: yes

  2. Referee: §5.1, Table 4: Hybrid markets lost to AI-only in 2 of 6 domains (marketing: 0.490 vs. 0.430; education: 0.488 vs. 0.458), which is one-third of tested domains. The abstract's phrase 'except for a few cases' understates this proportion. The framing should be made consistent with the actual results.

    Authors: We agree. Two out of six domains is one-third of the tested domains, and the phrase 'except for a few cases' in the abstract understates this proportion and implies a stronger result than the data support. We will revise the abstract and the corresponding language in Section 5.1 and the conclusions to accurately reflect that hybrid markets underperformed artificial-only markets in two of six domains. We will also ensure that the framing throughout the paper is consistent with the actual distribution of results rather than presenting a selectively optimistic characterization. revision: yes

  3. Referee: §5.1, Table 4: In 2 of the 3 domains where human-only baselines were run, human-only markets matched or outperformed hybrid markets (economics: 0.414 vs. 0.411; psychology: 0.378 vs. 0.523). This raises a direct question about whether AI agents add value or introduce noise relative to human-only markets. The paper does not address this finding.

    Authors: This is a fair and important observation that we did not adequately address in the manuscript. In psychology, the human-only market (MAE 0.378) substantially outperformed both the hybrid (0.523) and artificial-only (0.528) markets, and in economics, the human-only market (0.414) was essentially tied with the hybrid market (0.411). This does raise the question of whether AI agents add value or introduce noise relative to human-only markets in certain domains. We will add a dedicated discussion of this finding. Our preliminary interpretation is that in domains where human experts have strong domain-specific intuition (as may be the case in psychology, where many participants may have direct familiarity with the studies or the methodological norms of the field), AI agents trained on potentially less representative training data may introduce noise rather than complementary signal. In psychology, the training corpus was heavily weighted toward psychology studies (252 of 402 training studies), which may have led to overfitting or reduced generalization to the specific test claims. We will also note that with n=5 per domain, these comparisons are themselves subject to substantial sampling uncertainty. We agree that this finding complicates the central claim about the value of human-AI collaboration and needs to be discussed transparently. revision: yes

  4. Referee: §4.2: Several hyperparameters (lambda, liquidity, percent difference, initial agent cash, market duration) were tuned for the hybrid experiments, but the tuned values are not reported, and the sensitivity of results to these choices is not discussed. The specific values should be reported, and ideally a sensitivity analysis or at least a justification for the chosen configuration should be provided.

    Authors: The referee is correct that the omission of tuned hyperparameter values makes the results difficult to interpret and reproduce. We will add a table reporting the specific values of all tuned hyperparameters (lambda, liquidity, percent difference, initial agent cash, market duration) used in the hybrid experiments. We will also add justification for the chosen configuration, explaining the dual-metric tuning approach described in Section 4.1 (predictive accuracy on training data plus plausibility of agent participation patterns). Regarding sensitivity analysis: given the computational cost of re-running the full set of live hybrid experiments with human participants, a complete sensitivity analysis across all hyperparameter configurations is not feasible within the revision timeframe. However, we can and will report the results of a sensitivity analysis on the artificial-only markets, which do not require human participants, to characterize how performance varies with key hyperparameters. We will be transparent about the limitation that sensitivity of the hybrid (human-inclusive) results to these choices is not fully characterized. revision: partial

standing simulated objections not resolved
  • The fundamental limitation of small sample sizes (n=5 studies per discipline, 30 total) cannot be resolved within a revision. Running additional live hybrid markets with human participants requires recruiting domain experts, scheduling events, and waiting for replication outcomes, which is not feasible on a revision timeline. We can and will be transparent about this limitation and frame our conclusions as preliminary, but we cannot increase the sample size.

Circularity Check

0 steps flagged

No significant circularity; central empirical claim is tested on held-out data, not forced by construction.

full rationale

The paper's central claim is that hybrid human-AI prediction markets match or outperform artificial-only markets for replication forecasting. This claim is evaluated empirically on 30 held-out test studies (5 per discipline) whose replication outcomes were not available to participants at experiment time. The algorithmic agents are trained on 402 prior replication studies, and the test set is drawn from a different source (SCORE program) than the training set. While the paper relies on self-citation for the market architecture ([44], [50]) and feature extraction pipeline ([62]), these citations provide infrastructure (the market mechanism, the 41 features) rather than the empirical result itself. The hybrid-vs-artificial comparison is not circular by construction: the hybrid market's predictions depend on both agent behavior (trained on training data) and human trading behavior (independent human judgment), and the comparison metric (MAE against ground-truth replication outcomes) is computed on studies not used in training. The self-citations are standard methodological scaffolding, not load-bearing for the central empirical claim. The paper's weaknesses (small n=5 per discipline, no significance tests, mixed results across domains) are correctness/statistical-power concerns, not circularity. No step in the derivation chain reduces to its inputs by definition.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 1 invented entities

The free parameters are mostly hyperparameters of the artificial market, tuned on training data. The key ad hoc axiom is that n=5 per domain supports claims of consistency. No new physical entities or theoretical constructs are introduced; the platform is an engineering artifact.

free parameters (6)
  • lambda = not specified numerically; described as 'reduced' for hybrid experiments
    LMSR parameter controlling market liquidity; tuned to prevent premature convergence over 12-hour experiments (Section 4.2).
  • liquidity = not specified numerically
    Market liquidity parameter; tuned alongside lambda during training phase (Section 4.1).
  • initial agent cash = $500
    Chosen to ensure agents remain solvent throughout 12-hour experiment (Section 4).
  • human participant cash = $25 per market
    Chosen to balance budgetary considerations with incentivization (Section 4).
  • market duration = 43200 seconds (12 hours)
    Set to approximate temporal dynamics of traditional financial markets (Section 4).
  • percent difference = not specified
    Hyperparameter optimized during training phase (Section 4.1); value not reported.
axioms (4)
  • standard math Logarithmic Market Scoring Rule (LMSR) produces well-calibrated probability estimates from market prices.
    Foundational assumption of the market mechanism; invoked throughout Section 4 and in [44].
  • domain assumption 41 extracted features (statistical, bibliometric, semantic) are sufficient for algorithmic agents to meaningfully differentiate replication likelihood.
    Feature set from [62] used for agent instantiation; assumes these features capture relevant signal (Section 3, Figure 1).
  • domain assumption Human participants trade on epistemic beliefs about replicability rather than purely profit-maximizing strategies.
    Survey evidence supports this (Section 5.2), but it is an assumption about participant motivation that affects market aggregation quality.
  • ad hoc to paper 5 test studies per discipline provide a meaningful comparison of market performance.
    The paper draws conclusions about consistent performance from n=5 per domain without statistical tests (Tables 2-4).
invented entities (1)
  • Hybrid prediction market platform independent evidence
    purpose: Web server hosting pre-trained artificial market with API endpoints for agent and human trading
    The platform is described and used in live experiments; it is a software system, not a theoretical entity. Falsifiable through replication of experiments.

pith-pipeline@v1.1.0-glm · 15827 in / 2586 out tokens · 1258926 ms · 2026-07-05T17:12:44.383327+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 64 canonical work pages · 8 internal anchors

  1. [1]

    Abernethy, Y

    J. Abernethy, Y . Chen, and J. Wortman Vaughan. An optimization-based framework for automated market-making. InProceedings of the 12th ACM conference on Electronic commerce, pages 297–306, 2011

    work page 2011

  2. [2]

    Alansari and H

    A. Alansari and H. Luqman. Large language models hallucination: A comprehensive survey.arXiv preprint arXiv:2510.06265, 2025

    work page arXiv 2025

  3. [3]

    Altmejd, A

    A. Altmejd, A. Dreber, E. Forsell, J. Huber, T. Imai, M. Johannesson, M. Kirchler, G. Nave, and C. Camerer. Predicting the replicability of social science lab experiments.PloS one, 14(12):e0225826, 2019

    work page 2019

  4. [4]

    K. J. Arrow, R. Forsythe, M. Gorham, R. Hahn, R. Hanson, J. O. Ledyard, S. Levmore, R. Litan, P. Mil- grom, F. D. Nelson, G. R. Neumann, M. Ottaviani, T. C. Schelling, R. J. Shiller, V . L. Smith, E. Snow- berg, C. R. Sunstein, P. C. Tetlock, P. E. Tetlock, H. R. Varian, J. Wolfers, and E. Zitzewitz. The promise of prediction markets.Science, 320(5878):87...

    work page 2008

  5. [5]

    Bansal, B

    G. Bansal, B. Nushi, E. Kamar, W. S. Lasecki, D. S. Weld, and E. Horvitz. Beyond accuracy: The role of mental models in human-ai team performance. InProceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pages 2–11, 2019

    work page 2019

  6. [6]

    Barbu and N

    A. Barbu and N. Lay. Artificial prediction markets for lymph node detection. In2013 E-Health and Bioengineering Conference (EHB), pages 1–7. IEEE, 2013

    work page 2013

  7. [7]

    Barbu, N

    A. Barbu, N. Lay, and S. Mannor. An introduction to artificial prediction markets for classification. Journal of Machine Learning Research, 13(7), 2012

    work page 2012

  8. [8]

    J. E. Berg, F. D. Nelson, and T. A. Rietz. Prediction market accuracy in the long run.International Journal of Forecasting, 24(2):285–300, 2008

    work page 2008

  9. [9]

    C. F. Camerer, A. Dreber, E. Forsell, T.-H. Ho, J. Huber, M. Johannesson, M. Kirchler, J. Almenberg, A. Altmejd, T. Chan, et al. Evaluating replicability of laboratory experiments in economics.Science, 351(6280):1433–1436, 2016

    work page 2016

  10. [10]

    C. F. Camerer, A. Dreber, F. Holzmeister, T.-H. Ho, J. Huber, M. Johannesson, M. Kirchler, G. Nave, B. A. Nosek, T. Pfeiffer, et al. Evaluating the replicability of social science experiments in nature and science between 2010 and 2015.Nature Human Behaviour, 2(9):637–644, 2018

    work page 2010

  11. [11]

    L. B. Canonico, C. Flathmann, and N. McNeese. The wisdom of the market: Using human factors to design prediction markets for collective intelligence. InProceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 63, pages 1471–1475. SAGE Publications Sage CA: Los Angeles, CA, 2019

    work page 2019

  12. [12]

    S. P. Chandrashekar, D. Viganola, A. Dreber, M. Johannesson, T. Pfeiffer, A. Siegel, and G. Feldman. Using prediction markets and forecasting surveys to predict 28 replication outcomes of classic articles in social psychology and judgement and decision making.Royal Society Open Science, 13(1), 2026

    work page 2026

  13. [13]

    S. Chen, Y . Chen, Z. Li, Y . Jiang, Z. Wan, Y . He, D. Ran, T. Gu, H. Li, T. Xie, et al. Benchmarking large language models under data contamination: A survey from static to dynamic evaluation. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10091–10109, 2025

    work page 2025

  14. [14]

    Y . Chen, L. Fortnow, N. Lambert, D. M. Pennock, and J. Wortman. Complexity of combinatorial market makers. InProceedings of the 9th ACM Conference on Electronic Commerce, pages 190–199, 2008

    work page 2008

  15. [15]

    What the HellaSwag? On the Validity of Common-Sense Reasoning Benchmarks

    P. Chizhov, M. Nee, P.-C. Langlais, and I. P. Yamshchikov. What the hellaswag? on the validity of common-sense reasoning benchmarks.arXiv preprint arXiv:2504.07825, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    O. S. Collaboration. Estimating the reproducibility of psychological science.Science, 349(6251):aac4716, 2015

    work page 2015

  17. [17]

    Cowgill, J

    B. Cowgill, J. Wolfers, and E. Zitzewitz. Using prediction markets to track information flows: Evidence from google. Inamma, page 3, 2009

    work page 2009

  18. [18]

    DellaVigna and E

    S. DellaVigna and E. Vivalt. Forecasting social science: Evidence from 100 projects. Technical report, National Bureau of Economic Research, 2025

    work page 2025

  19. [19]

    The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems

    D. Dellermann, A. Calma, N. Lipusch, T. Weber, S. Weigel, and P. Ebel. The future of human- ai collaboration: a taxonomy of design knowledge for hybrid intelligence systems.arXiv preprint arXiv:2105.03354, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  20. [20]

    Dellermann, P

    D. Dellermann, P. Ebel, M. S ¨ollner, and J. M. Leimeister. Hybrid intelligence.Business & information systems engineering, 61(5):637–643, 2019

    work page 2019

  21. [21]

    T. M. Errington, E. Iorns, W. Gunn, F. E. Tan, J. Lomax, and B. A. Nosek. An open investigation of the reproducibility of cancer biology research.Elife, 3, 2014

    work page 2014

  22. [22]

    Fahse and A

    T. Fahse and A. Schmitt. Exploring the synergies in human-ai hybrids: A longitudinal analysis in sales forecasting. 2023

    work page 2023

  23. [23]

    S. S. Feger, S. Dallmeier-Tiessen, A. Schmidt, and P. W. Wo ´zniak. Designing for reproducibility: A qualitative study of challenges and opportunities in high energy physics. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2019

    work page 2019

  24. [24]

    Fogliato, M

    R. Fogliato, M. De-Arteaga, and A. Chouldechova. A case for humans-in-the-loop: Decisions in the presence of misestimated algorithmic scores.Available at SSRN 4050125, 2022

    work page 2022

  25. [25]

    Fraser, M

    H. Fraser, M. Bush, B. C. Wintle, F. Mody, E. T. Smith, A. M. Hanea, E. Gould, V . Hemming, D. G. Hamilton, L. Rumpff, et al. Predicting reliability through structured expert elicitation with the replicats (collaborative assessments for trustworthy science) process.Plos one, 18(1):e0274429, 2023

    work page 2023

  26. [26]

    F ¨ugener, J

    A. F ¨ugener, J. Grahl, A. Gupta, and W. Ketter. Will humans-in-the-loop become borgs? merits and pitfalls of working with ai.MIS quarterly, 45(3):1527–1556, 2021

    work page 2021

  27. [27]

    B. J. Gillen, C. R. Plott, and M. Shum. Information aggregation mechanisms in the field: Sales forecast- ing inside intel. InWorking paper. 2012

    work page 2012

  28. [28]

    Hanson, R

    R. Hanson, R. Oprea, and D. Porter. Information aggregation and manipulation in an experimental market.Journal of Economic Behavior & Organization, 60(4):449–459, 2006

    work page 2006

  29. [29]

    R. H. Harper. The role of hci in the age of ai.International Journal of Human–Computer Interaction, 35(15):1331–1344, 2019

    work page 2019

  30. [30]

    C. F. Horn, B. S. Ivens, M. Ohneberg, and A. Brem. Prediction markets: A literature review 2014.The Journal of Prediction Markets, 8(2):89–126, 2014

    work page 2014

  31. [31]

    Hu and A

    J. Hu and A. Storkey. Multi-period trading prediction markets with connections to machine learning. In International Conference on Machine Learning, pages 1773–1781, 2014

    work page 2014

  32. [32]

    Jahedpari, T

    F. Jahedpari, T. Rahwan, S. Hashemi, T. P. Michalak, M. De V os, J. Padget, and W. L. Woon. Online prediction via continuous artificial prediction markets.IEEE Intelligent Systems, 32(1):61–68, 2017

    work page 2017

  33. [33]

    J. Kim, A. Podlasek, K. Shidara, F. Liu, A. Alaa, and D. Bernardo. Limitations of large language models in clinical problem-solving arising from inflexible reasoning.Scientific reports, 15(1):39426, 2025

    work page 2025

  34. [34]

    R. A. Klein, K. A. Ratliff, M. Vianello, R. B. Adams Jr, ˇS. Bahn´ık, M. J. Bernstein, K. Bocian, M. J. Brandt, B. Brooks, C. C. Brumbaugh, et al. Investigating variation in replicability.Social psychology, 2014

    work page 2014

  35. [35]

    R. A. Klein, M. Vianello, F. Hasselman, B. G. Adams, R. B. Adams Jr, S. Alper, M. Aveyard, J. R. Axt, M. T. Babalola, ˇS. Bahn´ık, et al. Many labs 2: Investigating variation in replicability across samples and settings.Advances in Methods and Practices in Psychological Science, 1(4):443–490, 2018

    work page 2018

  36. [36]

    Lay and A

    N. Lay and A. Barbu. Supervised aggregation of classifiers using artificial prediction markets. InICML, 2010

    work page 2010

  37. [37]

    The Artificial Regression Market

    N. Lay and A. Barbu. The artificial regression market.arXiv preprint arXiv:1204.4154, 2012

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  38. [38]

    J. Li, I. Aziez, R. Boudreault, and D. Lafond. Augmented human-ai forecasting for ship refit project scheduling: A predict-then-optimize approach. InInternational Conference on Machine Learning, Op- timization, and Data Science, pages 375–389. Springer, 2024

    work page 2024

  39. [39]

    J. Li, Y . Yin, D. Lafond, A. Ghasemi, C. Diallo, and E. Bertrand. Integrated human-ai forecasting for preventive maintenance task duration estimation. InInternational Conference on Machine Learning, Optimization, and Data Science, pages 3–18. Springer, 2023

    work page 2023

  40. [40]

    Lin, L.-P

    V . Lin, L.-P. Morency, and E. Ben-Michael. Omitted variable bias in language models under distribution shift.arXiv preprint arXiv:2602.16784, 2026

    work page arXiv 2026

  41. [41]

    Y . Liu, M. Gordon, J. Wang, M. Bishop, Y . Chen, T. Pfeiffer, C. Twardy, and D. Viganola. Repli- cation markets: Results, lessons, challenges and opportunities in ai replication.arXiv preprint arXiv:2005.04543, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  42. [42]

    C. F. Manski. Interpreting the predictions of prediction markets.economics letters, 91(3):425–429, 2006

    work page 2006

  43. [43]

    Nair and M

    D. Nair and M. J. Saenz. Pair people and ai for better product demand forecasting.MIT Sloan Manage- ment Review, 65(2):1–5, 2024

    work page 2024

  44. [44]

    Design and Analysis of a Synthetic Prediction Market using Dynamic Convex Sets

    N. Nakshatri, A. Menon, C. L. Giles, S. Rajtmajer, and C. Griffin. Design and analysis of a synthetic prediction market using dynamic convex sets.arXiv preprint arXiv:2101.01787, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  45. [45]

    National Academies Press, 2019

    National Academies of Sciences, Engineering, and Medicine et al.Reproducibility and replicability in science. National Academies Press, 2019

    work page 2019

  46. [46]

    Okamura and S

    K. Okamura and S. Yamada. Adaptive trust calibration for human-ai collaboration.Plos one, 15(2):e0229132, 2020

    work page 2020

  47. [47]

    Pawel and L

    S. Pawel and L. Held. Probabilistic forecasting of replication studies.PloS one, 15(4):e0231416, 2020

    work page 2020

  48. [48]

    P. M. Polgreen, F. D. Nelson, G. R. Neumann, and R. A. Weinstein. Use of prediction markets to forecast infectious disease activity.Clinical Infectious Diseases, 44(2):272–279, 2007

    work page 2007

  49. [49]

    X. Puig, T. Shu, S. Li, Z. Wang, Y .-H. Liao, J. B. Tenenbaum, S. Fidler, and A. Torralba. Watch-and- help: A challenge for social perception and human-ai collaboration.arXiv preprint arXiv:2010.09890, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  50. [50]

    Rajtmajer, C

    S. Rajtmajer, C. Griffin, J. Wu, R. Fraleigh, L. Balaji, A. Squicciarini, A. Kwasnica, D. Pennock, M. McLaughlin, T. Fritton, et al. A synthetic prediction market for estimating confidence in published work. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 13218–13220, 2022

    work page 2022

  51. [51]

    Revilla, M

    E. Revilla, M. J. Saenz, M. Seifert, and Y . Ma. Human–artificial intelligence collaboration in prediction: a field experiment in the retail industry.Journal of Management Information Systems, 40(4):1071–1098, 2023

    work page 2023

  52. [52]

    Schoenegger, I

    P. Schoenegger, I. Tuminauskaite, P. S. Park, R. V . S. Bastos, and P. E. Tetlock. Wisdom of the sil- icon crowd: Llm ensemble prediction capabilities rival human crowd accuracy.Science Advances, 10(45):eadp1528, 2024

    work page 2024

  53. [53]

    Spann and B

    M. Spann and B. Skiera. Sports forecasting: a comparison of the forecast accuracy of prediction markets, betting odds and tipsters.Journal of Forecasting, 28(1):55–72, 2009

    work page 2009

  54. [54]

    Spiesberger, J

    A. Spiesberger, J. J. Vazquez, N. Pochinkov, T. Gaven ˇciak, P. Grietzer, G. Leech, and N. Schoots. Soft contamination means benchmarks test shallow generalization.arXiv preprint arXiv:2602.12413, 2026

    work page arXiv 2026

  55. [55]

    A. Storkey. Machine learning markets. InProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 716–724, 2011

    work page 2011

  56. [56]

    Isoelastic Agents and Wealth Updates in Machine Learning Markets

    A. Storkey, J. Millin, and K. Geras. Isoelastic agents and wealth updates in machine learning markets. arXiv preprint arXiv:1206.6443, 2012

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  57. [57]

    P. C. Tetlock. Liquidity and prediction market efficiency.Available at SSRN 929916, 2008

    work page 2008

  58. [58]

    G. V . Travaini, F. Pacchioni, S. Bellumore, M. Bosia, and F. De Micco. Machine learning and criminal justice: a systematic review of advanced methodology for recidivism risk prediction.International journal of environmental research and public health, 19(17):10594, 2022

    work page 2022

  59. [59]

    D. Wang, E. Churchill, P. Maes, X. Fan, B. Shneiderman, Y . Shi, and Q. Wang. From human-human collaboration to human-ai collaboration: Designing ai systems that can work together with people. In Extended abstracts of the 2020 CHI conference on human factors in computing systems, pages 1–6, 2020

    work page 2020

  60. [60]

    D. Wang, J. D. Weisz, M. Muller, P. Ram, W. Geyer, C. Dugan, Y . Tausczik, H. Samulowitz, and A. Gray. Human-ai collaboration in data science: Exploring data scientists’ perceptions of automated ai.Pro- ceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–24, 2019

    work page 2019

  61. [61]

    Wolfers and E

    J. Wolfers and E. Zitzewitz. Interpreting prediction market prices as probabilities. Technical report, National Bureau of Economic Research, 2006

    work page 2006

  62. [62]

    J. Wu, R. Nivargi, S. S. T. Lanka, A. M. Menon, S. A. Modukuri, N. Nakshatri, X. Wei, Z. Wang, J. Caverlee, S. M. Rajtmajer, et al. Predicting the reproducibility of social and behavioral science papers using supervised learning models.arXiv preprint arXiv:2104.04580, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  63. [63]

    Y . Yang, W. Youyou, and B. Uzzi. Estimating the deep replicability of scientific findings using human and artificial intelligence.Proceedings of the National Academy of Sciences, 117(20):10762–10768, 2020

    work page 2020

  64. [64]

    Youyou, Y

    W. Youyou, Y . Yang, and B. Uzzi. A discipline-wide investigation of the replicability of psychology pa- pers over the past two decades.Proceedings of the National Academy of Sciences, 120(6):e2208863120, 2023

    work page 2023