{"id":"ada147a1-006d-4f28-8f13-1ac4b54e1d9f","arxiv_id":"2506.05636","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A Bayesian latent-correlation model with entropy-based query selection predicts expert majority votes while querying fewer human experts than two baseline methods on four image classification tasks.","lead":"A new Bayesian model treats human experts and a pretrained classifier as correlated voters, then decides which experts to ask and when to stop asking on each image. On four medical and crowdsourced image datasets, it reaches the expert majority vote with fewer average queries than two comparison methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The query-savings claim depends on the Gaussian logit model (Sec. 4, Eq. 2); non-Gaussian or class-conditional logit distributions can break the calibration of the stopping rule, and the reported ECE does not cover the e=0 operating points used for the headline query counts.","rationale":"I read the paper in good faith. The method is coherent, the code is public, and the experiments show a consistent direction. The reader's CONDITIONAL verdict is appropriate. My stress-test agrees with the reader's weakest assumption: the stationary multivariate normal coupling in Eq. (2) is the least secure premise. I want to make the attack sharp: the stopping rule is the load-bearing mechanism, and it inherits all calibration risk from the Gaussian model. The paper's own admission that predictive distributions were initially miscalibrated and required a temperature patch is evidence that the Gaussian model is not automatically calibrated. The ECE evidence is reassuring but limited; it is measured on the adaptive stream and not at the zero-error operating points used to report the headline query counts. I did not find an internal inconsistency in the derivation of Eq. (4); Algorithm 1 correctly implements the importance-weighted Monte Carlo. The baselines are admittedly modified, which weakens the comparison, but the absolute query counts (e.g., 2.55 vs. 5 for ChestX-ray) still support the method's value. The proposed concrete test -- replacing the Gaussian with a mixture and evaluating on a held-out stream -- would directly settle whether the Gaussian assumption is what drives the query savings. Until that test is run, CONDITIONAL is the right verdict.","tokens_in":73,"tokens_out":17750,"duration_ms":250727,"concrete_test":"Replicate the ChestX-ray and Chaoyang experiments replacing Eq. (2)'s single multivariate normal with a two-component Gaussian mixture over logits (or a Student-t copula with matched marginals), keeping the query-selection and stopping rules identical. Then compare the average number of queries at 0% error and the ECE at e=0.01 on a strictly held-out 250-example stream that was not used for any posterior update. If either the query savings shrink to within run-to-run standard error or ECE degrades by more than 1%, the Gaussian assumption is load-bearing for the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim -- that the Bayesian querying scheme reaches 0% consensus error with fewer expert queries than baselines -- rests on the stopping rule in Sec. 6, which halts when the model's posterior error estimate falls below threshold e. That estimate is computed from the posterior predictive in Eq. (4), which is built on the assumption in Eq. (2) that the vector of classifier logits and latent human logits is jointly multivariate normal with a single global mean mu and covariance Sigma. If the true distribution of these logits is non-Gaussian (e.g., class-conditional multimodality, heavy tails, or nonlinear expert-classifier agreement), the model's posterior predictive can be miscalibrated: it may be overconfident exactly in regions where the expert consensus is uncertain, causing premature stopping and errors that the ECE table in Sec. 7.4.1 does not exclude. The temperature parameter tau in Eq. (3) only rescales the softmax probabilities uniformly; it cannot repair a misspecified conditional distribution p(z_H | z_M), yet it is the only mechanism introduced to fix miscalibration. Moreover, the ECE values in Table 1 are computed on the same online streams used for model updates and only at thresholds e in {0.05, 0.025, 0.01}; the headline 0%-error query counts correspond to the e->0 endpoint, where calibration is not measured. Because the query-count differences against baseline (2.55 vs 3.16 and 1.58 vs 1.82) are small, a modest calibration failure could invert the comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a Bayesian framework for predicting the consensus labels of a panel of human experts, using classifier probability estimates and partially observed expert votes. The generative model places a multivariate normal distribution on joint classifier and latent human logits, with a global temperature on the vote likelihood. Inference is performed via MCMC, and an expected-entropy criterion selects which expert to query next; querying stops when the model's posterior error estimate falls below a threshold. Experiments on ChestX-ray, Chaoyang, CIFAR-10H, and ImageNet-16H compare the method with two modified baselines and report error-cost curves, ECE values, exploration/exploitation behavior, and a distribution-shift adaptation.","tokens_in":18951,"tokens_out":12267,"duration_ms":117251,"significance":"The paper addresses a practically important and under-explored problem: minimizing expert query cost while predicting expert consensus. The model is general with respect to aggregation functions and identifiable experts, and the authors provide code and data. The empirical ECE results at the tested thresholds and the distribution-shift experiment are useful. However, the central claim of query savings rests on comparisons with same-group modified baselines and on average query counts without variability measures, so the current evidence is suggestive rather than definitive.","major_comments":[{"comment":"The headline comparison (2.55 vs 3.16 queries for ChestX-ray and 1.58 vs 1.82 for Chaoyang) is presented as averaged over 12 runs, but no standard errors, confidence intervals, or paired tests are reported. The two differences (0.61 and 0.24 queries) are small relative to the likely run-to-run variability of an online procedure on 250-example streams, so without significance information the claim that the method uses fewer queries on average is not statistically supported. Please report per-run values or error bars and a paired significance test for the endpoint query counts.","section":"7.3, Fig. 2"},{"comment":"Both baselines are modifications of methods from papers with overlapping authorship (Showalter et al. 2024 and Kerrigan et al. 2021). The modifications are non-trivial: INFEXP is augmented with epsilon-greedy expert selection, and the confusion-matrix method is redefined with respect to consensus. No comparison is made to the original methods, to random querying, or to simple uncertainty-sampling policies. This makes it difficult to attribute the observed gains to the Bayesian generative model rather than to baseline implementation choices. At minimum, include simple querying baselines and, if feasible, independent implementations or the original authors' code.","section":"7.2, Baselines"},{"comment":"The vote likelihood is not consistently defined. gamma maps probability vectors to logits (Eq. 1), and z_i are defined as logits, but Eq. (3) defines TS(theta_i,tau)=softmax(theta_i/tau) for a probability vector theta_i, whereas Algorithm 1 evaluates TS(gamma(z_i^s),tau), which is undefined as written because gamma(z_i^s) is not defined for a K-1 dimensional logit vector. This ambiguity affects the likelihood in Eq. (4) and the simulated votes in Algorithms 1 and 2. Please define the inverse logistic transformation explicitly and state whether the temperature is applied to logits (softmax(z/tau)) or to probability vectors.","section":"Section 4, Eq. (3); Algorithm 1"},{"comment":"The paper's main operating point is the threshold (or smoothed-curve point) at which 0% error is reached, but the ECE table covers only e in {0.05, 0.025, 0.01}. If the 0%-error query counts are obtained from lowess-smoothed curves or from thresholds outside this range, the reported ECEs do not establish calibration at those operating points. The Gaussian logit assumption in Eq. (2) is strong, and the temperature parameter in Eq. (3) cannot repair conditional misspecification, so calibration at the exact stopping thresholds used for the headline numbers is the load-bearing quantity. Please state which thresholds produce the reported 0%-error points, report raw unsmoothed values, and provide ECE or actual error rates at those same thresholds.","section":"7.3 and 7.4.1, Table 1"}],"minor_comments":[{"comment":"The sentence 'our method consistently minimizes the error rate (i.e., reaches 0% error)' is ambiguous for CIFAR-10H and ImageNet-16H, where the text states that both baselines never reach 0% error. Please clarify whether the proposed method reaches 0% on all four datasets and, if not, restrict the 0%-error claim to the two medical datasets or rephrase.","section":"Section 7.3, last paragraph"},{"comment":"If the reported query counts for the 0%-error points are read from lowess-smoothed curves rather than from raw averaged data, this should be stated explicitly; smoothed values can cross zero at query counts where no actual run achieved zero error.","section":"Section 7.3, Fig. 2"},{"comment":"The statement that the confusion+calibration baseline never reaches 0% error even when all experts are queried is surprising, since with all votes observed the consensus is known; please clarify the mechanism (e.g., a MAP over a latent consensus model disagreeing with the observed majority) or report the baseline's error when forced to output the observed majority.","section":"Section 7.3"},{"comment":"The notation M is used both for the number of classifiers and for the set of classifier indices; please use a different symbol (e.g., a calligraphic M) for the set. Also, Appendix D reports a single set of hyperparameters; since these control the exploration/exploitation trade-off and hence query counts, a sensitivity analysis would help establish robustness of the query-count differences.","section":"Section 3 and Appendix D"},{"comment":"The distribution-shift experiment reports a 0% error rate after the shift, but no error bars or comparison to a non-adaptive baseline are provided; this would strengthen the claim that the sliding-window adaptation is responsible for the maintained accuracy.","section":"Section 7.4.2, Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The overlap between the authors and the originators of both baselines is a concern for the impartiality of the comparison; I would encourage the editor to consider whether independent baseline implementations or third-party evaluation are needed. The paper is well within scope for the journal; the methodology is sound and the topic is timely, but the empirical evidence is not yet conclusive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick verdict: this is a real piece of work, a clean extension of the authors' INFEXP framework to identifiable, correlated experts with per-expert query selection. The Bayesian derivation in Appendix A is coherent, the code and data are public, and the ECE numbers suggest the stopping rule is reasonably calibrated at the thresholds they test. The paper deserves a serious referee.\n\nWhat's new: Showalter et al. (2024) treated experts as exchangeable and non-identifiable, deciding only how many to query. Here each expert gets their own latent logit, the joint distribution over classifier and expert logits is modeled with a multivariate normal, and the query policy picks the expert that minimizes expected posterior entropy of the consensus. That's a genuine step forward, and the framework also handles alternative aggregation functions like any-positive or unanimous. The experiments on two medical datasets plus CIFAR-10H/ImageNet-16H all point in the same direction.\n\nSoft spots. First, both baselines are modified by the authors, one from their own group (INFEXP) and one from Kerrigan et al., which also has overlapping authorship. The modifications are described carefully, but there's no independent baseline, and the reported savings are small—2.55 vs 3.16 queries on ChestX-ray, 1.58 vs 1.82 on Chaoyang. A modest calibration failure could invert those differences. Second, the error-cost curves have no error bars; the 12-run averages are fine, but the spread matters when the gaps are small. Third, the ECE table covers thresholds 0.05, 0.025, and 0.01, but the headline 0%-error query counts correspond to the e→0 endpoint, where calibration is not measured. Fourth, the model assumes the joint logit vector is multivariate normal with a single global mean and covariance. The temperature parameter τ rescales the softmax but cannot repair a misspecified conditional distribution p(z_H | z_M). The low ECE values at the tested thresholds are reassuring, but they don't fully settle the non-Gaussian case. Finally, there's a notation slip in Eq. (3): it writes TS(θ_i, τ) with θ_i a probability vector, but the softmax should act on the logits γ(θ_i); the algorithm and appendices use the latter, so it's a typo rather than a substantive error.\n\nBottom line: the central method is sound, the empirical direction is consistent, and the weaknesses are addressable. This paper is for people working on human-AI collaboration, active learning, and crowdsourced labeling. I'd send it to peer review, with a request for error bars, ECE at the e→0 operating point, and preferably at least one independent baseline.","headline":"Legitimate extension of INFEXP to identifiable experts; query-savings claim is credible but the evidence needs error bars, ECE at the actual operating point, and an independent baseline.","tokens_in":19503,"tokens_out":3131,"would_cite":true,"duration_ms":28426,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","68T05","68T37"],"pacs":[],"model":"deepseek-v4-flash","headline":"A Bayesian model predicts expert consensus using fewer expert queries.","keywords":["Bayesian inference","expert consensus prediction","human-AI collaboration","active querying","multivariate normal model","logistic normal","calibration","MCMC"],"falsifier":"Build a dataset whose classifier logits are bimodal, heavy-tailed, or generated from a mixture of two image populations with different expert-classifier correlations, run the method with the paper's hyperparameters and threshold $e=0.01$, and check whether the realized consensus error stays at 0% and whether the expected calibration error stays below 1%: if measured error rises above the threshold or calibration degrades while simpler baselines retain their error levels, the Gaussian assumption is the cause.","tokens_in":18415,"feed_emoji":"🩻","tokens_out":4745,"duration_ms":49036,"temperature":0.7,"pith_summary":"The paper claims that a Bayesian generative model over correlated classifier outputs and human expert votes can decide online, per image, which experts to query and when to stop, so that the group's majority label is predicted perfectly with fewer expert queries than existing methods. On two real medical datasets the method reaches zero consensus-prediction error with on average 2.55 versus 3.16 queries (ChestX-ray) and 1.58 versus 1.82 queries (Chaoyang) compared with the strongest baseline. The practical point is that expensive expert time can be saved whenever a cheap pretrained classifier is informative about how a panel of identifiable experts will vote, even when the classifier was trained independently of those experts.","feed_headline":"Bayesian model predicts expert consensus with fewer queries","feed_subtitle":"On chest X-rays it hits 0 error in 2.55 queries vs 3.16 for the baseline, saving expert time.","key_machinery":"The load-bearing object is the joint multivariate normal distribution over the additive-logistic-transformed logits of all agents, $z \\sim N(\\mu, \\Sigma)$, with a global temperature $\\tau$ that turns latent probability vectors into observed votes. The query policy is expected-entropy reduction of the consensus posterior, evaluated by simulating each candidate expert's vote and re-running the conditional sampling of the unobserved votes. This Gaussian prior is what lets correlations among identifiable experts and the black-box classifier be learned from a small online stream, and the temperature is what recalibrates the vote likelihood so that posterior error estimates match realized error.","core_discovery":"The central claim is that expert votes can be treated as noisy categorical draws from latent per-expert probability vectors whose logits, together with classifier logits, follow one multivariate normal with a shared mean and covariance, learned online via MCMC. Conditioned on this model, the expected entropy of the consensus distribution after each hypothetical vote identifies the most informative expert to query next, and querying stops when the model's consensus-error estimate falls below a threshold. The paper reports that this procedure consistently achieves zero error on the expert-consensus task across ChestX-ray, Chaoyang, CIFAR-10H, and ImageNet-16H, and does so with fewer queries on average than the INFEXP and confusion-plus-calibration baselines, while maintaining expected calibration error generally below one percent.","pith_inferences":["Because the covariance dimension grows as $(K-1)(M+H)$, scaling to many classes or many agents will require low-rank or factored covariance approximations; the paper's Gaussian core would still drive query selection if such structure were imposed.","The expected-entropy query rule is a greedy information-gain heuristic; a testable extension would compare it to non-greedy lookahead or to an oracle that queries the expert whose vote changes the consensus most often under the posterior, to see how much of the savings comes from the Gaussian correlation model versus the query rule.","The method implicitly assumes the pretrained classifier's predicted probabilities carry signal about the panel's votes; a stress test would swap in a classifier that is calibrated but negatively correlated with experts and measure whether query savings persist.","Since the method maintains a posterior over each identifiable expert's latent propensity, it could double as a diagnostic for expert disagreement patterns, flagging images where a single expert is likely to dissent from consensus."],"forward_implications":["On the medical datasets, the method reaches 0% consensus error with 2.55 queries versus 3.16 (ChestX-ray) and 1.58 versus 1.82 (Chaoyang) compared with INFEXP with epsilon-greedy querying, so expert panels can be consulted less often without losing fidelity to the panel's majority.","Because the model's error estimates are calibrated (ECE generally below 1%), the error threshold $e$ is a practical dial: setting $e=0.01$ yields actual error near zero and allows per-example stopping.","The same trained model directly supports aggregation functions other than majority vote, such as 'any expert votes positive' or 'all experts agree,' by changing the function $f$, with measurably different query costs.","In a distribution-shift experiment on ImageNet-16H, a sliding window of recent examples makes the model increase exploration when accuracy drops, preserving a 0% error rate despite the shift.","The framework does not require the classifier to have been trained on expert labels, so it applies when the classifier and experts are mismatched."],"supporting_citations":[{"why":"Supplies the closest prior method and the INFEXP baseline, which decides how many experts to query without modeling which expert to query.","marker":"Showalter et al. (2024)"},{"why":"Provides the confusion-plus-calibration baseline, extended here to multiple experts and consensus prediction.","marker":"Kerrigan et al. (2021)"},{"why":"Contributes the ChestX-ray dataset with five identifiable radiologist labels used in the medical evaluation.","marker":"Nabulsi et al. (2021)"},{"why":"Contributes the Chaoyang colon-image dataset with three pathologist labels and the ResNet classifier used in the evaluation.","marker":"Zhu et al. (2021)"},{"why":"Provides CIFAR-10H human annotations, which the paper combines into synthetic identifiable experts.","marker":"Peterson et al. (2019)"},{"why":"Provides ImageNet-16H annotations and the Bayesian human-AI complementarity modeling that motivates correlating humans and classifiers.","marker":"Steyvers et al. (2022)"},{"why":"Supplies the DenseNet chest X-ray classifier whose probabilities are used as the black-box model output for ChestX-ray.","marker":"Tang et al. (2020)"},{"why":"Establishes the logistic-normal distribution that maps probability vectors to real logits for the multivariate normal prior.","marker":"Atchison & Shen (1980)"},{"why":"Models correlations among probabilistic classifiers via multivariate normal mixtures, an approach the paper adapts to include human votes.","marker":"Pirš & Štrumbelj (2019)"},{"why":"Provides the LKJ prior on correlation matrices used for the covariance of the joint logit distribution.","marker":"Lewandowski et al. (2009)"}],"fun_headline_variants":["Bayesian model nails expert consensus in fewer queries","Fewer expert queries, zero-error consensus via Bayesian inference","Bayesian querying cuts expert calls while keeping accuracy","Zero-error consensus with fewer human queries: Bayesian approach","Bayesian expert model: high accuracy, low query cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the assumption that, after the logistic transform, the joint vector of classifier confidences and latent expert confidences is Gaussian with one mean and covariance shared by every image, plus a single vote temperature; if the real joint distribution is non-Gaussian, changes over time, or needs different parameters for different image types, the model's error estimates become miscalibrated and the reported query savings are not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian model nails expert consensus in fewer queries","Fewer expert queries, zero-error consensus via Bayesian inference","Bayesian querying cuts expert calls while keeping accuracy","Zero-error consensus with fewer human queries: Bayesian approach","Bayesian expert model: high accuracy, low query cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000783,"raw_usage":{"total_tokens":3394,"prompt_tokens":822,"completion_tokens":2572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":2495}},"tokens_in":438,"tokens_out":2572,"duration_ms":19984,"temperature":1.0,"reasoning_tokens":2495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:13:56.076895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a dataset whose classifier logits are bimodal, heavy-tailed, or generated from a mixture of two image populations with different expert-classifier correlations, run the method with the paper's hyperparameters and threshold $e=0.01$, and check whether the realized consensus error stays at 0% and whether the expected calibration error stays below 1%: if measured error rises above the threshold or calibration degrades while simpler baselines retain their error levels, the Gaussian assumption is the cause.","supporting_citations":[{"cited_title":"J., Smyth, P., and Steyvers, M","cited_arxiv_id":null,"evidence_quote":"Supplies the closest prior method and the INFEXP baseline, which decides how many experts to query without modeling which expert to query."},{"cited_title":"Combining human predictions with model probabilities via confusion matrices and calibration","cited_arxiv_id":null,"evidence_quote":"Provides the confusion-plus-calibration baseline, extended here to multiple experts and consensus prediction."},{"cited_title":"P., Ye, W., Yang, J., Pilgrim, R., Kazemzadeh, S., et al","cited_arxiv_id":null,"evidence_quote":"Contributes the ChestX-ray dataset with five identifiable radiologist labels used in the medical evaluation."},{"cited_title":"Hard sample aware noise robust learning for histopathology image classification","cited_arxiv_id":null,"evidence_quote":"Contributes the Chaoyang colon-image dataset with three pathologist labels and the ResNet classifier used in the evaluation."},{"cited_title":"C., Battleday, R","cited_arxiv_id":null,"evidence_quote":"Provides CIFAR-10H human annotations, which the paper combines into synthetic identifiable experts."},{"cited_title":"Bayesian modeling of human– AI complementarity","cited_arxiv_id":null,"evidence_quote":"Provides ImageNet-16H annotations and the Bayesian human-AI complementarity modeling that motivates correlating humans and classifiers."},{"cited_title":"A., Brandon, C","cited_arxiv_id":null,"evidence_quote":"Supplies the DenseNet chest X-ray classifier whose probabilities are used as the black-box model output for ChestX-ray."}],"review_version":1}