Pith. sign in

REVIEW 5 major objections 8 minor 159 references

A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection

T0 review · 5 major / 8 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A class-conditional Rasch model recovers rare labels in crowdsourcing better than standard aggregators by letting annotator skill and item difficulty differ by class.

desk verdict Solid methods paper that fills the DS–GLAD gap for minority recall; the F1 drop on high-recall wins is real and under-discussed, but the operational claim is still coherent. read the letter →

arxiv 2607.24622 v1 pith:DET53S7X submitted 2026-07-27 stat.ML cs.LG

Gabriel Singer , Samuel Gruffaz , Olivier Vo Van , Nicolas Vayatis , Argyris Kalogeratos This is my paper
classification stat.MLcs.LG
keywords crowdsourcinglabelaggregationclassimbalanceminorityrecallRaschmodelannotatorcompetenceitemdifficultyCondorcetjurytheorem
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When the labels that matter most are also the rarest, majority vote and common aggregation methods can look accurate overall while missing the minority class. This paper argues that the missing piece is a model in which both annotator competence and item difficulty are allowed to depend on the true class, so the crowd can contain majority specialists, minority specialists, and people who are good or bad on both. It introduces CC-Rasch, a generative model that does exactly that, proves a Condorcet-style guarantee and a class-proportion preservation result for majority vote under imbalance, and shows on 33 real crowdsourcing datasets that the model consistently leads in minority recall while staying competitive on balanced accuracy. The practical claim is simple: if rare-label recovery is the goal, class-dependent ability and difficulty should be estimated jointly rather than ignored or collapsed into a single skill score.

What carries the argument

CC-Rasch: a class-conditional Rasch-style generative model where Z_ir | Y_i=k ~ Bernoulli(σ(α_{r,k}−β_{i,k})), with α_{r,k}=μ_{α,k}+g_{r,k} and β_{i,k}=μ_{β,k}+h_{i,k}, sum-to-zero constraints, Gaussian priors, and EM estimation; it is the object that jointly captures class-dependent competence and difficulty.

What would settle it

On a held-out imbalanced crowdsourcing task where confusions are highly asymmetric (for example systematic majority-to-minority bias) or difficulty is not scalar per class, check whether CC-Rasch still leads minority recall against Dawid–Skene and PLAT; a clear loss under those conditions would undercut the model’s central claim.

Watch

Extended reading notes

Core claim

CC-Rasch, which models correctness probability as a logistic of class-specific annotator ability minus class-specific item difficulty (with mixed-effect deviations and uniform error spreading over wrong classes), recovers minority labels more reliably than majority vote, Dawid–Skene, GLAD, CrowdFM, and PLAT across a large real-world benchmark, while remaining competitive in balanced accuracy.

Load-bearing premise

Given the true class, every wrong label is treated as equally likely, and correctness is fully summarized by one logistic skill-minus-difficulty number per class.

Editorial extensions

If this is right

  • Inspection and quality systems that care most about rare defects can prioritize CC-Rasch-style aggregation when only crowd labels are available.
  • Majority vote remains consistent for every class, including minorities, only if each class has enough specialists relative to weak or adversarial annotators.
  • As the number of labels grows, majority vote’s implied class balance converges to the true balance under the paper’s crowd conditions.
  • Gold labels are not always required for strong minority recall: the unsupervised class-conditional model can beat gold-supervised baselines as imbalance grows.
  • Benchmarking label aggregation under imbalance should treat minority recall as primary rather than overall accuracy alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If minority specialists are scarce in a deployed crowd, the paper’s synthetic sweep suggests actively recruiting or weighting them may matter more than adding generic annotators.
  • The uniform-error assumption points to a natural next model: class-to-class confusion structure combined with class-specific difficulty.
  • The same mixed-effects split could transfer to other noisy-label settings where both rater skill and case hardness are known to be label-dependent, such as medical triage or content moderation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper studies label aggregation under class imbalance, where rare classes are the operationally important ones. It makes three contributions: (1) a class-imbalanced Condorcet Jury Theorem (Theorem 3.1) giving crowd-composition conditions under which majority vote asymptotically recovers every class, plus a corollary that MV asymptotically preserves the class proportions; (2) CC-Rasch, a generative model with class-dependent annotator abilities α_{r,k} = μ_{α,k} + g_{r,k} and class-dependent item difficulties β_{i,k} = μ_{β,k} + h_{i,k}, Gaussian priors on the deviations, sum-to-zero constraints, and uniform error spreading over the remaining K−1 classes (Eq. 8), fitted by penalized EM with a proved unique M-step maximizer (Proposition 4.2); (3) an evaluation on 33 real crowdsourcing datasets plus a synthetic benchmark, showing the highest average minority recall among six methods while remaining competitive in balanced accuracy, with the largest gains under strong imbalance, high annotation redundancy, and large-scale regimes. The theory, as sketched, is plausible: Theorem 3.1 reduces to a per-class application of a cited proposition, Corollary 3.2 is a total-variation argument, and Proposition 4.2 follows from strict concavity of the penalized Q-function plus coercivity. My concerns are mainly empirical-methodological (metric choice versus the paper's own per-dataset F1 numbers, and dataset accounting) and the dependence of Theorem 3.1 on an unpublished companion paper.

Significance. If the empirical claims hold up, the paper is a useful contribution: a principled generative aggregator for the practically important and under-studied imbalanced-crowdsourcing regime, with a uniqueness guarantee for the EM M-step, an unusually broad real-data evaluation (binary, multiclass, and two large-scale regimes), and a well-designed gold-supervision comparison in §5.3. The imbalanced Condorcet condition and the imbalance-preservation corollary are interpretable and extend prior work to heterogeneous multiclass crowds. The main risks are (i) that part of the minority-recall advantage is an operating-point artifact invisible to the chosen metrics, (ii) dataset-accounting errors that may inflate the benchmark's size and bias aggregates, and (iii) the key jury-theorem lemma living in an unpublished companion paper. All are addressable within the manuscript's scope.

major comments (5)
  1. [§5, Tables 1–4] The headline claim is 'highest minority recall while remaining competitive in balanced accuracy' (Abstract; Tables 1–3). Both metrics are recall-only: neither penalizes false positives on the majority class, so an aggregator that predicts the minority class more often inflates both. The paper's own Table 4 shows this signature: product (recall 0.825 vs DS 0.640 but Macro F1 0.707 vs 0.843), Loneliness Older Adults (F1 0.494 vs MV 0.667 at equal minority recall), Jigsaw (F1 0.918 vs DS 0.982), Amazon Sentiment Negative (F1 0.762 vs 0.90x for all baselines). Table 3, the only aggregate table with F1, has CC-Rasch below DS (0.812/0.793 vs 0.831/0.830). Per-dataset balanced accuracy is not reported, so the aggregate 'competitive' claim may hide per-dataset collapses. Please (a) report per-class precision or Macro F1 in aggregate Tables 1–2 alongside recall, (b) report per-dataset balanced ac
  2. [§5.2, Table 4] The '33 datasets' claim needs an audit. §5's text says 'we evaluate our CC-Rasch model across 27 real-world datasets' while the Abstract and Table 4 say 33. Table 4 contains 'bird' (Imb 1.28) and 'Bird' (Imb 1.25) as separate rows with otherwise identical statistics — apparently the same dataset counted twice. Weather Sentiment–AMT (Imb 10.22, 20.00 ann./item) has values numerically identical to CF* (10.22, 20.10) in every row, strongly suggesting a copy-paste error. CF and CF* share Imb 10.22. Since the benchmark's breadth is a stated contribution ('one of the largest benchmarks'), please provide a definitive dataset table (source, K, n, annotators, Imb), reconcile the 27/33 discrepancy, and correct or justify the duplicated rows, since duplicated near-balanced datasets would bias the aggregate means in Tables 1–2.
  3. [§3, Theorem 3.1] Theorem 3.1 is presented as a multiclass, multidimensional generalization, but the proof reduces entirely to 'Proposition D.3 of Singer et al. (2026) applies to each class k' — a lemma in the appendix of the authors' own unpublished preprint, presumably stated for the binary case. Two gaps: (i) For K > 2, the MV error event given Y = k is that some other class out-polls k; reducing this to a single binary 'good vs bad annotator' condition per class requires at minimum a union bound over the K−1 competing classes, which is not shown, and the pairwise margin needed depends on the runner-up class probabilities, not just p_r^(k). (ii) The condition uses the binary-flavored threshold p_r^(k) > 1/2 + δ_k, which is not the natural competence threshold for K > 2 (random guessing is 1/K). Please make the multiclass reduction explicit with the union-bound argument, and include a self-contained pro
  4. [§4.2, §5 (Implementation details)] The penalized EM has several tuning quantities — prior variances σ²_{α,k}, σ²_{β,k}, the mean penalties λ_{α,μ}, λ_{β,μ}, initialization of (G, H), number of EM restarts, and convergence criteria — none of which are reported in §5 (only 'L-BFGS-B with warm start' and uniform π init). Note also that Proposition 4.2 holds π fixed, so it is unclear whether/how π is updated, which matters under imbalance. Since the empirical comparison is the paper's central claim, please report these settings, a sensitivity analysis over the prior variances (the penalty both regularizes and, via coercivity, drives the uniqueness result), and state whether code will be released.
  5. [§4.2, Eq. (8)] Eq. (8) spreads annotation errors uniformly over the other K−1 classes, so the model captures class-dependent correctness but not class-dependent confusion structure. This is load-bearing for the model's motivation: if real annotators exhibit systematic majority→minority confusion (plausibly common in inspection tasks), the likelihood mis-weights exactly the minority specialists the model is designed to find. The synthetic benchmark (§5.1) is binary with symmetric error flips, so this failure mode is never exercised. Please add a synthetic multiclass experiment with an asymmetric confusion structure (e.g., a Dawid–Skene-style generator with majority→minority bias) showing whether CC-Rasch's recall advantage survives, and discuss the limitation explicitly.
minor comments (8)
  1. [Table 4] Table 4 names the proposed method 'Φ-Rasch (OUR)' while the rest of the paper calls it CC-Rasch (also '(OUR)' vs '(OURS)' across tables). Please unify.
  2. [§2–§4] Notation inconsistencies: §2 uses m annotators / n items; §3 uses R annotators with set RR; Proposition 4.1 says 'for any r ∈ [n]' (should be the annotator index set); §4.2 'for all (i,r) ∈ Ω_r × [R]' is malformed (Ω_r undefined; presumably Ω_i × ... or O); Corollary 3.2 uses max over 'k ∈ K' instead of [K].
  3. [§5.2, Table 2] PLAT is absent from the multiclass results (Table 2) without explanation; if it is binary-only, say so where baselines are introduced.
  4. [§4.1, Proposition 4.1] Proposition 4.1 (GLAD sensitivity = specificity) is stated without proof; a two-line derivation from Eq. (5) would suffice.
  5. [Throughout] Typos/formatting: 'on33real-world', 'a stronge class imbalance', 'one need to estimate', 'compare it performances', 'Wherem denotes', missing spaces in references ('PauliVirtanen'), 'École Polytechnique' citation style.
  6. [§1, Contributions] Contributions list says 'four baselines' but five are compared (MV, DS, GLAD, CrowdFM, PLAT).
  7. [§5.2, Table 4] In Table 4 most rows report ±0.000 across 4 seeds, suggesting deterministic fitting; please state which methods are seed-dependent and why variance vanishes, otherwise the '4 random seeds' protocol is unclear.
  8. [§5.1, Figure 1] Figure 1's dual x-axes (minority % decreasing left-to-right on top, majority % increasing on bottom) are easy to misread; consider plotting recall vs p_min only.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity: Condorcet result leans on one same-author lemma, but the model and minority-recall claims are estimated and scored against external ground truth.

  1. self citation load bearing [§3 Theorem 3.1 proof]
    "Under condition (7), Proposition D.3 of Singer et al. (2026) applies to each class k∈[K], yielding lim R→∞ eR(k)=0. ... This is a generalization to the multidimension and multiclass case of Proposition D.3 in Singer et al. (2026)."

    The only non-elementary step in the imbalanced Condorcet argument is imported from a concurrent same-author paper rather than derived here. The present theorem is essentially 'apply our prior binary proposition class-wise under (7).' This is mild: condition (7) is stated independently, the reduction is explicit, and the central CC-Rasch/empirical claims do not depend on this lemma. Not a fit-as-prediction or definitional loop.

full rationale

The paper's load-bearing empirical claim (CC-Rasch highest minority recall on 33 datasets) is not circular: parameters are fit by penalized EM on observed crowd labels only, and minority recall / balanced accuracy are computed against held-out ground-truth labels that never enter the likelihood. The generative model (class-conditional logistic correctness, uniform error spread, Gaussian mixed effects, sum-to-zero constraints) is an ansatz, not a quantity defined from the evaluation metrics. Proposition 4.1 (GLAD sensitivity=specificity) and Proposition 4.2 (unique M-step maximizer via strict concavity + coercivity on a closed convex set) are self-contained derivations from the stated equations. Theorem 3.1 and Corollary 3.2 state crowd-composition hypotheses and deduce asymptotic consistency and proportion preservation; they are not fitted to the tables. The only mild circularity-adjacent step is that the multiclass Condorcet proof invokes Proposition D.3 of Singer et al. (2026)—same author list—rather than re-proving the binary building block in-line. That citation is not load-bearing for the model or the empirical minority-recall claim, and does not force the reported metrics by construction. Score 1.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The central empirical claim rests on the CC-Rasch likelihood and EM fitting; the theory rests on conditional independence of annotators and competence lower bounds. Free parameters are the usual IRT/EM variances and class proportions. No exotic physical entities; invented pieces are modeling constructs standard in psychometrics/crowdsourcing.

free parameters (3)
  • Prior variances σ²_{α,k}, σ²_{β,k} (and optional μ penalties)
    Gaussian mixed-effect priors fix scale/translation; values are not reported as data-driven fits in the main text but control regularization strength of g_{r,k}, h_{i,k}.
  • Class proportions π_k = uniform initialization
    Initialized uniformly and used in the complete-data likelihood / E-step; can be updated or held as part of the generative model.
  • Synthetic reliability table q_{t,k}, hard-item penalty δ, imbalance π, type proportions = p_good=p_bad=25%, p_maj+p_min=50%, Imb≈7
    Hand-chosen generative parameters for the controlled specialist sweep in §5.1 / Fig. 1.
assumptions (5)
  • domain assumption Annotator labels are conditionally independent given the true item label Y.
    Stated explicitly before Theorem 3.1; standard crowdsourcing assumption used for both jury theorem and likelihood factorization.
  • ad hoc to paper P(correct | Y=k) = σ(α_{r,k} − β_{i,k}) with errors uniform on the remaining classes (Eq. 8).
    Core CC-Rasch observation model; extends Rasch/GLAD but the uniform multiclass error spread is a modeling choice not forced by data.
  • ad hoc to paper Sum-to-zero constraints ∑_r g_{r,k}=∑_i h_{i,k}=0 and independent Gaussian priors on deviations.
    Identifiability / mixed-effects regularization device in §4.2, following Whitehill/Liu-style practice.
  • domain assumption For Condorcet: liminf crowd composition inequality (7) with class-wise specialist sets G^{(k)} and bad sets B^{(k)}.
    Hypothesis of Theorem 3.1; if violated (too few minority specialists), consistency need not hold.
  • standard math Logistic link and Bernoulli correctness indicators as in classical Rasch/GLAD.
    Inherited item-response machinery (Rasch 1960; Whitehill et al. 2009).
invented entities (2)
  • CC-Rasch (class-conditional mixed-effects Rasch) parameter set (μ_α, μ_β, G, H)
    purpose: Jointly represent per-class annotator ability and per-class item difficulty for latent-label EM aggregation.
    New model class relative to single-ability GLAD and confusion-matrix Dawid–Skene; still a parametric statistical model, not a new physical object.
  • Minority-class / majority-class specialist partitions G^{(j_min)}, G^{(j_max)} independent evidence
    purpose: Decompose the crowd for imbalanced Condorcet analysis and synthetic robustness sweeps.
    Definitional subsets based on p^{(k)}_r thresholds; useful taxonomy rather than a latent discovered substance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection." pith.science (2026). https://pith.science/paper/DET53S7X

@misc{pith2026260724622,
  author       = {Pith},
  title        = {Pith review of: A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DET53S7X}},
  note         = {Machine review of arXiv:2607.24622}
}
abstract

We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones. In this setting, annotators may be reliable on both classes, unreliable on both classes, majority-class specialists, or minority-class specialists. Existing models only partially address this problem: they either capture class-dependent errors but ignore item difficulty, or they model item difficulty without capturing class-dependent errors. To fill this gap for imbalanced datasets in crowdsourcing, we introduce a generative aggregation model combining item difficulty with class-dependent annotator competence. The model allows both annotator abilities and item difficulties to vary across classes. We then revisit Condorcet's Jury Theorem in the class-imbalanced setting. We also show that majority voting asymptotically preserves the underlying class proportion. We evaluate our model on $33$ real-world crowdsourcing datasets, covering multiclass tasks such as images and text, as well as two large-scale regimes: large-scale annotation datasets, with many annotations per item, and large-scale item datasets, with a large number of annotated instances. Across these diverse settings, our model consistently achieves the highest minority recall while remaining competitive in balanced accuracy, making it particularly relevant when rare-label recovery is the primary objective.

Figures

Figures reproduced from arXiv: 2607.24622 by the authors.

Figure 1
Figure 1. Minority-class recall as the proportion of majority-class specialists [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Gold label boosted methods versus CC-Rasch. [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

159 extracted references · 4 canonical work pages

  1. [1]

    , title =

    Whitehill, Jacob and Ruvolo, Paul and Wu, Ting-fan and Bergsma, Jacob and Movellan, Javier R. , title =. Advances in Neural Information Processing Systems , volume =

  2. [2]

    Intelligent Data Analysis , volume =

    Japkowicz, Nathalie and Stephen, Shaju , title =. Intelligent Data Analysis , volume =. 2002 , doi =

    2002

  3. [3]

    and Jordan, Michael I

    Wainwright, Martin J. and Jordan, Michael I. , title =. Foundations and Trends in Machine Learning , volume =. 2008 , doi =

    2008

  4. [4]

    , title =

    Nguyen, Duc and Zhang, Anderson Y. , title =. Advances in Neural Information Processing Systems , volume =

  5. [5]

    Rasch, Georg , title =

  6. [6]

    and Laird, Nan M

    Dempster, Arthur P. and Laird, Nan M. and Rubin, Donald B. , title =. Journal of the Royal Statistical Society: Series B (Methodological) , volume =. 1977 , doi =

    1977

  7. [7]

    and Steinhardt, Jacob , title =

    Lipton, Zachary C. and Steinhardt, Jacob , title =. Queue , volume =. 2019 , publisher =

    2019

  8. [8]

    and Chiang, Mung and Studer, Christoph , title =

    Lan, Andrew S. and Chiang, Mung and Studer, Christoph , title =. Proceedings of the 35th International Conference on Machine Learning , series =

Show all 159 references
  1. [9]

    and Plan, Yaniv and van den Berg, Ewout and Wootters, Mary , title =

    Davenport, Mark A. and Plan, Yaniv and van den Berg, Ewout and Wootters, Mary , title =. Information and Inference: A Journal of the IMA , volume =. 2014 , doi =

    2014

  2. [10]

    , title =

    Andersen, Erling B. , title =. Journal of the Royal Statistical Society: Series B (Methodological) , volume =. 1970 , doi =

    1970

  3. [11]

    Statistics & Probability Letters , volume =

    Ghosh, Malay , title =. Statistics & Probability Letters , volume =. 1995 , doi =

    1995

  4. [12]

    Haberman , title =

    Shelby J. Haberman , title =. The Annals of Statistics , number =. 1977 , doi =

    1977

  5. [13]

    Wainwright and Bin Yu , title =

    Sivaraman Balakrishnan and Martin J. Wainwright and Bin Yu , title =. The Annals of Statistics , number =. 2017 , doi =

    2017

  6. [14]

    2019 , eprint=

    Class-Balanced Loss Based on Effective Number of Samples , author=. 2019 , eprint=

    2019

  7. [15]

    2026 , eprint=

    MIFair: A Mutual-Information Framework for Intersectionality and Multiclass Fairness , author=. 2026 , eprint=

    2026

  8. [16]

    1997 , publisher=

    Spectral Graph Theory , author=. 1997 , publisher=

    1997

  9. [17]

    2023 , eprint=

    Fairness Guarantee in Multi-Class Classification , author=. 2023 , eprint=

    2023

  10. [18]

    International Conference on Machine Learning , pages=

    Fair and optimal classification via post-processing , author=. International Conference on Machine Learning , pages=

  11. [19]

    Preprint arXiv:2202.09724 , year=

    Bayes-optimal classifiers under group fairness , author=. Preprint arXiv:2202.09724 , year=

    arXiv

  12. [20]

    2022 , eprint=

    Blackbox Post-Processing for Multiclass Fairness , author=. 2022 , eprint=

    2022

  13. [21]

    Proceedings of 2nd Workshop on Diversity in Artificial Intelligence (AIDBEI) , year =

    Characterizing Intersectional Group Fairness with Worst-Case Comparisons , author =. Proceedings of 2nd Workshop on Diversity in Artificial Intelligence (AIDBEI) , year =

  14. [22]

    Shuman, D. I. and Narang, S. K. and Frossard, P. and Ortega, A. and Vandergheynst, P. , year=. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , volume=. IEEE Signal Processing Magazine , publi...

    2012

  15. [23]

    Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining , pages=

    Algorithmic decision making and the cost of fairness , author=. Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining , pages=

  16. [24]

    Fairness definitions explained , doi =

    Verma, Sahil and Rubin, Julia , year =. Fairness definitions explained , doi =

  17. [25]

    2018 , eprint=

    Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness , author=. 2018 , eprint=

    2018

  18. [26]

    2025 , eprint=

    Beyond Flatland: A Geometric Take on Matching Methods for Treatment Effect Estimation , author=. 2025 , eprint=

    2025

  19. [27]

    2011 , eprint=

    Fairness Through Awareness , author=. 2011 , eprint=

    2011

  20. [28]

    The distance function and Lipschitz classes of mappings between metric spaces , volume =

    Markovic, Marijan , year =. The distance function and Lipschitz classes of mappings between metric spaces , volume =. Mathematika , doi =

  21. [29]

    CoRR , volume =

    Moritz Hardt and Eric Price and Nathan Srebro , title =. CoRR , volume =. 2016 , url =. 1610.02413 , timestamp =

    2016 arXiv

  22. [30]

    Algorithmic decision making and the cost of fairness , journal =

    Sam Corbett. Algorithmic decision making and the cost of fairness , journal =. 2017 , url =. 1701.08230 , timestamp =

    2017 arXiv

  23. [31]

    Journal of Machine Learning Research , year =

    Hilde Weerts and Miroslav DudÃk and Richard Edgar and Adrin Jalali and Roman Lutz and Michael Madaio , title =. Journal of Machine Learning Research , year =

  24. [32]

    The Risk of Racial Bias in Hate Speech Detection

    Sap, Maarten and Card, Dallas and Saeidi, Saeid and Kleinberg, Adam and Jurafsky, Dan and Smith, Noah A. The Risk of Racial Bias in Hate Speech Detection. 57th Annual Meeting of the Association for Computational Linguistics. 2019

    2019

  25. [33]

    Consistency of weighted majority votes , volume =

    Berend, Daniel and Kontorovich, Aryeh , booktitle =. Consistency of weighted majority votes , volume =

  26. [34]

    Advances in Neural Information Processing Systems , volume=

    The multidimensional wisdom of crowds , author=. Advances in Neural Information Processing Systems , volume=

  27. [35]

    Advances in Neural Information Processing Systems , volume=

    Whose vote should count more: Optimal integration of labels from labelers of unknown expertise , author=. Advances in Neural Information Processing Systems , volume=

  28. [36]

    World Wide Web Conference , pages=

    Nuanced metrics for measuring unintended bias with real data for text classification , author=. World Wide Web Conference , pages=

  29. [37]

    International Economic Review , volume=

    Optimal decision rules in uncertain dichotomous choice situations , author=. International Economic Review , volume=. 1982 , publisher=

    1982

  30. [38]

    Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing , pages=

    Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets , author=. Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing , pages=

  31. [39]

    Preprint arXiv:2110.04052 , year=

    On Release! 50 Annotator Labels of 10K Toxicity Data , author=. Preprint arXiv:2110.04052 , year=

    arXiv

  32. [40]

    International Conference on Machine Learning , pages =

    A reductions approach to fair classification , author =. International Conference on Machine Learning , pages =

  33. [41]

    Advances in Neural Information Processing Systems , volume =

    Empirical risk minimization under fairness constraints , author =. Advances in Neural Information Processing Systems , volume =

  34. [42]

    2019 , eprint=

    Human uncertainty makes classification more robust , author=. 2019 , eprint=

    2019

  35. [43]

    Learning with Noisy Labels Revisited: A Study on

    Wei, Jiaheng and Zhu, Zhaowei and Cheng, Hao and Liu, Tongliang and Niu, Gang and Liu, Yang , booktitle=. Learning with Noisy Labels Revisited: A Study on

  36. [44]

    Maximum Likelihood Estimation of Observer Error-Rates Using the

    Dawid, Alexander Philip and Skene, Allan M , journal=. Maximum Likelihood Estimation of Observer Error-Rates Using the. 1979 , publisher=

    1979

  37. [45]

    Civil Engineering and Environmental Systems , volume=

    Evaluation of inspection features including exposure risk using a value of information analysis , author=. Civil Engineering and Environmental Systems , volume=. 2021 , publisher=

    2021

  38. [46]

    Preprint arXiv:2210.06812 , year =

    CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators , author =. Preprint arXiv:2210.06812 , year =

    arXiv

  39. [47]

    ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages=

    Towards fair truth discovery from biased crowdsourced answers , author=. ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages=

  40. [48]

    Test , volume=

    Reducing degradation and age of items in imperfect repair modeling , author=. Test , volume=. 2022 , publisher=

    2022

  41. [50]

    Advances in Neural Information Processing Systems , volume =

    Counterfactual fairness , author =. Advances in Neural Information Processing Systems , volume =

  42. [51]

    2023 , school =

    Uncovering Bias and Fairness in Crowdsourced Labeling , author =. 2023 , school =

    2023

  43. [52]

    International Conference on Machine Learning , pages =

    Fairness-aware learning for continuous attributes and treatments , author =. International Conference on Machine Learning , pages =

  44. [53]

    2018 , publisher=

    Lectures on the Poisson process , author=. 2018 , publisher=

    2018

  45. [54]

    International Conference on Machine Learning , pages=

    Gaussian process classification and active learning with multiple annotators , author=. International Conference on Machine Learning , pages=

  46. [55]

    2019 , publisher=

    Multilevel modeling , author=. 2019 , publisher=

    2019

  47. [56]

    IEEE Access , volume=

    Mixed-effects nonhomogeneous Poisson process model for multiple repairable systems , author=. IEEE Access , volume=. 2021 , publisher=

    2021

  48. [57]

    , author =

    Learning from crowds. , author =. Journal of Machine Learning Research , volume =

  49. [58]

    Jigsaw unintended bias in toxicity classification , howpublished =

  50. [59]

    Knowledge and Information Systems , volume=

    Data preprocessing techniques for classification without discrimination , author=. Knowledge and Information Systems , volume=. 2012 , publisher=

    2012

  51. [60]

    Science advances , volume=

    The accuracy, fairness, and limits of predicting recidivism , author=. Science advances , volume=. 2018 , publisher=

    2018

  52. [61]

    AAAI/ACM Conference on AI, Ethics, and Society , pages=

    Crowdsourcing with fairness, diversity and budget constraints , author=. AAAI/ACM Conference on AI, Ethics, and Society , pages=

  53. [62]

    Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages =

    Bayesian crowdsourcing with constraints , author =. Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages =. 2021 , organization =

    2021

  54. [63]

    Preprint arXiv:2005.00582 , year =

    Learning to complement humans , author =. Preprint arXiv:2005.00582 , year =

    2005 arXiv

  55. [64]

    Conference on Causal Learning and Reasoning , pages =

    Fair classification with instance-dependent label noise , author =. Conference on Causal Learning and Reasoning , pages =

  56. [65]

    Pattern Recognition , pages =

    Learning to Complement with Multiple Humans , author =. Pattern Recognition , pages =. 2025 , publisher =

    2025

  57. [66]

    2021 , eprint=

    Binary classification with corrupted labels , author=. 2021 , eprint=

    2021

  58. [67]

    Classification in the Presence of Label Noise: A Survey , year=

    Frenay, Benoit and Verleysen, Michel , journal=. Classification in the Presence of Label Noise: A Survey , year=

  59. [68]

    ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

    To aggregate or not? learning with separate noisy labels , author=. ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

  60. [69]

    and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , year=

    Selvaraju, Ramprasaath R. and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , year=. International Journal of Computer Vision , publisher=

  61. [70]

    Raykar and Shipeng Yu and Linda H

    Vikas C. Raykar and Shipeng Yu and Linda H. Zhao and Gerardo Hermosillo Valadez and Charles Florin and Luca Bogoni and Linda Moy , title =. Journal of Machine Learning Research , year =

  62. [71]

    5th International Conference on Learning Representations , year =

    Jacob Goldberger and Ehud Ben-Reuven , title =. 5th International Conference on Learning Representations , year =

  63. [72]

    Using Boosting to Detect Noisy Data

    Wheway, Virginia. Using Boosting to Detect Noisy Data. Advances in Artificial Intelligence. PRICAI 2000 Workshop Reader. 2001

    2000

  64. [73]

    Machine Learning , year=

    Angluin, Dana and Laird, Philip , title=. Machine Learning , year=

  65. [74]

    2022 , eprint=

    Learning from Noisy Labels with Deep Neural Networks: A Survey , author=. 2022 , eprint=

    2022

  66. [75]

    Advances in Neural Information Processing Systems 30 (NeurIPS 2017) , pages =

    Eran Malach and Shai Shalev-Shwartz , title =. Advances in Neural Information Processing Systems 30 (NeurIPS 2017) , pages =

    2017

  67. [76]

    MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels , booktitle =

    Lu Jiang and Zhengyuan Zhou and Thomas Leung and Li. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels , booktitle =

  68. [77]

    Tsang and Masashi Sugiyama , title =

    Bo Han and Quanming Yao and Xingrui Yu and Gang Niu and Miao Xu and Weihua Hu and Ivor W. Tsang and Masashi Sugiyama , title =. Advances in Neural Information Processing Systems (NeurIPS) , pages =

  69. [78]

    Aritra Ghosh and Naresh Manwani and P. S. Sastry , title =. AAAI Conference on Artificial Intelligence , pages =

  70. [79]

    Advances in Neural Information Processing Systems (NeurIPS) , pages =

    Zhilu Zhang and Murat Sabuncu , title =. Advances in Neural Information Processing Systems (NeurIPS) , pages =

  71. [80]

    Conference on Empirical Methods in Natural Language Processing , pages=

    Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks , author=. Conference on Empirical Methods in Natural Language Processing , pages=

  72. [81]

    IEEE International Conference on Computer Vision , pages =

    Yisen Wang and Xingjun Ma and Zaiyi Chen and Yuan Luo and James Bailey and Jinfeng Yi , title =. IEEE International Conference on Computer Vision , pages =

  73. [82]

    2015 , eprint=

    Training Convolutional Networks with Noisy Labels , author=. 2015 , eprint=

    2015

  74. [83]

    Tsang and Ya Zhang and Masashi Sugiyama , title =

    Bo Han and Jiangchao Yao and Gang Niu and Mingyuan Zhou and Ivor W. Tsang and Ya Zhang and Masashi Sugiyama , title =. Advances in Neural Information Processing Systems (NeurIPS) , pages =

  75. [84]

    2019 , eprint=

    Learning to Reweight Examples for Robust Deep Learning , author=. 2019 , eprint=

    2019

  76. [85]

    Advances in Neural Information Processing Systems (NeurIPS) , pages =

    Jun Shu and Qi Xie and Linyi Yi and Qian Zhao and Shu-Tao Zhou and Zongben Xu and Deheng Meng , title =. Advances in Neural Information Processing Systems (NeurIPS) , pages =

  77. [86]

    Junnan Li and Richard Socher and Steven C. H. Hoi , title =. International Conference on Learning Representations , year =

  78. [87]

    CVPR , year=

    Learning from Massive Noisy Labeled Data for Image Classification , author=. CVPR , year=

  79. [88]

    R. R. Bahadur and R. Ranga Rao , title =. The Annals of Mathematical Statistics , number =

  80. [89]

    International Conference on Learning Representations , year =

    Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations , author =. International Conference on Learning Representations , year =

  81. [90]

    IEEE Conference on Computer Vision and Pattern Recognition , pages =

    Learning from Massive Noisy Labeled Data for Image Classification , author =. IEEE Conference on Computer Vision and Pattern Recognition , pages =

  82. [91]

    Li, Wen and Wang, Limin and Li, Wei and Agustsson, Eirikur and Van Gool, Luc , journal =

  83. [92]

    Song, Hwanjun and Kim, Minseok and Lee, Jae-Gil , booktitle =

  84. [93]

    Learning Multiple Layers of Features from Tiny Images , author =

  85. [94]

    Proceedings of the IEEE , volume =

    Gradient-based learning applied to document recognition , author =. Proceedings of the IEEE , volume =

  86. [95]

    Xiao, Han and Rasul, Kashif and Vollgraf, Roland , journal =

  87. [96]

    NIPS Workshop on Deep Learning and Unsupervised Feature Learning , year =

    Reading digits in natural images with unsupervised feature learning , author =. NIPS Workshop on Deep Learning and Unsupervised Feature Learning , year =

  88. [97]

    Dua, Dheeru and Graff, Casey , howpublished =

  89. [98]

    2021 , eprint=

    Analysis of Boolean Functions , author=. 2021 , eprint=

    2021

  90. [99]

    A Sharp Uniform Bound for the Distribution of Sums of Bernoulli Trials , volume=

    Baillon, Jean-Bernard and Cominetti, Roberto and Vaisman, José , year=. A Sharp Uniform Bound for the Distribution of Sums of Bernoulli Trials , volume=. Combinatorics, Probability and Computing , publisher=

  91. [100]

    Journal of Inequalities and Applications , year=

    A nonuniform local limit theorem for Poisson binomial random variables via Stein's method , author=. Journal of Inequalities and Applications , year=

  92. [101]

    Proceedings of ICEET , year =

    Thermal Imaging of Utility Power Lines: A Review , author =. Proceedings of ICEET , year =

  93. [102]

    Journal of Risk and Reliability , year =

    Decision-theoretic inspection planning using imperfect and incomplete data , author =. Journal of Risk and Reliability , year =

  94. [103]

    Preprint arXiv:2301.00067 , year =

    Convolutional Non-Homogeneous Poisson Process with Application to Wildfire Risk Quantification for Power Delivery Networks , author =. Preprint arXiv:2301.00067 , year =

    arXiv

  95. [104]

    Communications in Statistics--Simulation and Computation , year =

    Reducing degradation and age of items in imperfect repair modeling , author =. Communications in Statistics--Simulation and Computation , year =

  96. [105]

    Procedia CIRP , volume =

    A Roadmap for Automated Power Line Inspection, Maintenance and Repair , author =. Procedia CIRP , volume =. 2013 , doi =

    2013

  97. [106]

    Reliability Engineering and System Safety , volume =

    Use of Dynamic Bayesian Networks for Life Extension Assessment of Ageing Systems , author =. Reliability Engineering and System Safety , volume =. 2015 , doi =

    2015

  98. [107]

    Journal of Machine Learning Research , volume =

    Fairness guarantees in multi-class classification with demographic parity , author =. Journal of Machine Learning Research , volume =

  99. [108]

    Advances in Neural Information Processing Systems , volume =

    The fragility of fairness: Causal sensitivity analysis for fair machine learning , author =. Advances in Neural Information Processing Systems , volume =

  100. [109]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

  101. [110]

    International Conference on Machine Learning , pages =

    Exact exponent in optimal rates for crowdsourcing , author =. International Conference on Machine Learning , pages =

  102. [111]

    Dempster, A. P. and Laird, N. M. and Rubin, D. B. , journal =. Maximum Likelihood from Incomplete Data via the

  103. [112]

    Advances in Neural Information Processing Systems , volume =

    Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model , author =. Advances in Neural Information Processing Systems , volume =

  104. [113]

    Reliability Engineering & System Safety , volume=

    Use of dynamic Bayesian networks for life extension assessment of ageing systems , author=. Reliability Engineering & System Safety , volume=. 2015 , publisher=

    2015

  105. [114]

    Chemical Engineering Transactions , volume=

    The pitfalls of risk based inspection: an insurer’s view of the Chevron Richmond and Tesoro Anacortes incidents , author=. Chemical Engineering Transactions , volume=

  106. [115]

    Data-Centric Engineering , volume=

    Decision-theoretic inspection planning using imperfect and incomplete data , author=. Data-Centric Engineering , volume=. 2021 , publisher=

    2021

  107. [116]

    Advances in Neural Information Processing Systems , volume =

    Learning from noisy labels via conditional distributionally robust optimization , author =. Advances in Neural Information Processing Systems , volume =

  108. [117]

    ACM Journal on Responsible Computing , volume =

    Bias mitigation for machine learning classifiers: A comprehensive survey , author =. ACM Journal on Responsible Computing , volume =. 2024 , publisher =

    2024

  109. [118]

    IEEE Signal Processing Magazine , volume =

    Learning from crowdsourced noisy labels: A signal processing perspective , author =. IEEE Signal Processing Magazine , volume =. 2025 , publisher =

    2025

  110. [119]

    Preprint arXiv:2304.12573 , year =

    Fairness and bias in truth discovery algorithms: An experimental analysis , author =. Preprint arXiv:2304.12573 , year =

    arXiv

  111. [120]

    Advances in Neural Information Processing Systems , volume =

    Noisy label learning with instance-dependent outliers: Identifiability via crowd wisdom , author =. Advances in Neural Information Processing Systems , volume =

  112. [121]

    Preprint arXiv:1805.11505 , year =

    Classification with imperfect training labels , author =. Preprint arXiv:1805.11505 , year =

    arXiv

  113. [122]

    Kennedy and Geoff Bacon and Alexander Sahn and Claudia von Vacano , title =

    Chris J. Kennedy and Geoff Bacon and Alexander Sahn and Claudia von Vacano , title =. Preprint arXiv: , year =

  114. [123]

    Biometrika , volume =

    Cannings, Timothy I and Fan, Yingying and Samworth, Richard J , title =. Biometrika , volume =. 2020 , month =. doi:10.1093/biomet/asaa011 , url =

    2020 doi

  115. [124]

    ChatGPT outperforms crowd workers for text-annotation tasks , volume=

    Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Maël , year=. ChatGPT outperforms crowd workers for text-annotation tasks , volume=. Proceedings of the National Academy of Sciences , publisher=. doi:10.1073/pnas.2305016120 , number=

    doi

  116. [125]

    2024 , isbn =

    Zeng, Xia and La Barbera, David and Roitero, Kevin and Zubiaga, Arkaitz and Mizzaro, Stefano , title =. 2024 , isbn =. doi:10.1145/3626772.3657965 , booktitle =

    2024

  117. [126]

    2024 , eprint=

    A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation , author=. 2024 , eprint=

    2024

  118. [127]

    2024 , eprint=

    AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators , author=. 2024 , eprint=

    2024

  119. [128]

    2026 , eprint=

    Optimal Fair Aggregation of Crowdsourced Noisy Labels using Demographic Parity Constraints , author=. 2026 , eprint=

    2026

  120. [129]

    2015 , eprint=

    Hoeffding's inequality for sums of weakly dependent random variables , author=. 2015 , eprint=

    2015

  121. [130]

    2025 , eprint=

    Fairness Amidst Non-IID Graph Data: A Literature Review , author=. 2025 , eprint=

    2025

  122. [131]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    Exploiting Worker Correlation for Label Aggregation in Crowdsourcing , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor =

    2019

  123. [132]

    Workshops Held at the First AAAI Conference on Human Computation and Crowdsourcing: A Report , volume =

    Josephy, Tatiana and Lease, Matt and Paritosh, Praveen and Krause, Markus and Georgescu, Mihai and Tjalve, Michael and Braga, Daniela , year =. Workshops Held at the First AAAI Conference on Human Computation and Crowdsourcing: A Report , volume =. Ai Magazine , doi =

  124. [133]

    2013 , issue_date =

    Rodrigues, Filipe and Pereira, Francisco and Ribeiro, Bernardete , title =. 2013 , issue_date =. doi:10.1016/j.patrec.2013.05.012 , journal =

    2013 doi

  125. [134]

    ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-Scale Entity Linking , journal =

    Demartini, Gianluca and Difallah, Djellel and Cudre-Mauroux, Philippe , year =. ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-Scale Entity Linking , journal =

  126. [135]

    The Multidimensional Wisdom of Crowds , url =

    Welinder, Peter and Branson, Steve and Perona, Pietro and Belongie, Serge , booktitle =. The Multidimensional Wisdom of Crowds , url =

  127. [136]

    2014 , eprint=

    Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing , author=. 2014 , eprint=

    2014

  128. [137]

    Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

    Snow, and Rion, and O'Connor, Brendan and Jurafsky, Daniel and Ng, Andrew , year =. Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. EMNLP’08 , journal =

  129. [138]

    Consistency of weighted majority votes , url =

    Berend, Daniel and Kontorovich, Aryeh , booktitle =. Consistency of weighted majority votes , url =

  130. [139]

    ACM Transactions on Knowledge Discovery from Data (TKDD) , volume=

    Exploiting heterogeneous graph neural networks with latent worker/task correlation information for label aggregation in crowdsourcing , author=. ACM Transactions on Knowledge Discovery from Data (TKDD) , volume=. 2021 , publisher=

    2021

  131. [140]

    Journal of Machine Learning Research , volume=

    Gradient descent for sparse rank-one matrix completion for crowd-sourced aggregation of sparsely interacting workers , author=. Journal of Machine Learning Research , volume=

  132. [141]

    2019 , eprint=

    Truth Inference at Scale: A Bayesian Model for Adjudicating Highly Redundant Crowd Annotations , author=. 2019 , eprint=

    2019

  133. [142]

    2014 , eprint=

    Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing , author=. 2014 , eprint=

    2014

  134. [143]

    Proceedings of the 35th International Conference on Machine Learning , pages =

    Crowdsourcing with Arbitrary Adversaries , author =. Proceedings of the 35th International Conference on Machine Learning , pages =. 2018 , editor =

    2018

  135. [144]

    Science China Information Sciences , volume=

    FNNWV: farthest-nearest neighbor-based weighted voting for class-imbalanced crowdsourcing , author=. Science China Information Sciences , volume=. 2024 , publisher=

    2024

  136. [145]

    arXiv preprint arXiv:1111.6554 , year=

    On the absolute constants in the Berry-Esseen type inequalities for identically distributed summands , author=. arXiv preprint arXiv:1111.6554 , year=

    arXiv

  137. [146]

    Journal of Applied Probability , author=

    Modelling dependence in simple and indirect majority systems , volume=. Journal of Applied Probability , author=. 1989 , pages=. doi:10.2307/3214318 , number=

    1989 doi

  138. [147]

    Monotonicity in Condorcet's Jury Theorem with dependent voters , urldate =

    Daniel Berend and Luba Sapir , journal =. Monotonicity in Condorcet's Jury Theorem with dependent voters , urldate =

  139. [148]

    When is Condorcet's Jury Theorem valid? , volume =

    Berend, Daniel and Paroush, Jacob , year =. When is Condorcet's Jury Theorem valid? , volume =. Social Choice and Welfare , doi =

  140. [149]

    Advances in Neural Information Processing Systems , volume=

    Learning stochastic majority votes by minimizing a PAC-Bayes generalization bound , author=. Advances in Neural Information Processing Systems , volume=

  141. [150]

    Peng Cao and Yilun Xu and Yuqing Kong and Yizhou Wang , booktitle=. Max-. 2019 , url=

    2019

  142. [151]

    doi:10.5281/zenodo.7024894 , url =

    Lara Orlandic and Tomas Teijeiro and David Atienza , title =. doi:10.5281/zenodo.7024894 , url =

    doi

  143. [152]

    and Torralba, Antonio and Murphy, Kevin P

    Russell, Bryan C. and Torralba, Antonio and Murphy, Kevin P. and Freeman, William T. , journal =

  144. [153]

    2024 , journal =

    Ustalov, Dmitry and Pavlichenko, Nikita and Tseitlin, Boris , title =. 2024 , journal =. doi:10.21105/joss.06227 , issn =. 2109.08584 , eprinttype =

    2024 arXiv

  145. [154]

    , journal=

    Zhang, Jing and Wu, Xindong and Sheng, Victor S. , journal=. Imbalanced Multiple Noisy Labeling , year=

  146. [155]

    2021 , eprint=

    Learning from Crowds with Sparse and Imbalanced Annotations , author=. 2021 , eprint=

    2021

  147. [156]

    The Fourteenth International Conference on Learning Representations , year=

    Towards a Foundation Model for Crowdsourced Label Aggregation , author=. The Fourteenth International Conference on Learning Representations , year=

  148. [157]

    AAAI Conference on Human Computation and Crowdsourcing , author=

    Modeling Temporal Crowd Work Quality with Limited Supervision , volume=. AAAI Conference on Human Computation and Crowdsourcing , author=. 2015 , pages=

    2015

  149. [158]

    Technologies , volume =

    Learning from Crowds Using a Focal Loss Function: Dealing with Imbalanced Annotations , author =. Technologies , volume =

  150. [159]

    and Haberland, Matt and Reddy, Tyler and Cournapeau, David and Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and Bright, Jonathan and

    Virtanen, Pauli and Gommers, Ralf and Oliphant, Travis E. and Haberland, Matt and Reddy, Tyler and Cournapeau, David and Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and Bright, Jonathan and. Nature Methods , year =

  151. [160]

    Analyse num

    Allaire, Gr. Analyse num. 2012 , publisher=

    2012

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.