Pith. sign in

REVIEW 2 major objections 5 minor 102 references

Jointly optimizing ensemble weights, size, and a signed diversity regularizer improves calibration to annotator disagreement by 40–78% while keeping competitive F1.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 06:33 UTC pith:HGZURVAO

load-bearing objection Practical prediction-space ensemble that delivers real CE/BS gains on subjective NLP by jointly learning weights, size and signed diversity; the fixed s choice is the main caveat the authors already flag. the 2 major comments →

arxiv 2607.08493 v1 pith:HGZURVAO submitted 2026-07-09 cs.LG cs.CL

Ensemble Diversity Optimization for Subjective Supervision

Xia Cui , Ziyi Huang , N. R. Abeynayake This is my paper
classification cs.LG cs.CL
keywords ensemble diversitysubjective supervisionannotator disagreementprobabilistic calibrationGumbel-Softmaxsoft labelsmulti-objective optimizationprediction-space learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Subjective NLP tasks produce systematic annotator disagreement that should be treated as signal rather than noise. This paper introduces Ensemble Diversity Optimization (EDO), a prediction-space method that learns ensemble weights and effective size end-to-end while a validation-tuned signed diversity term either preserves or suppresses disagreement. The joint multi-objective loss balances a soft F1 surrogate, class-weighted cross-entropy, and reliability-weighted diversity so the ensemble neither collapses nor overfits dominant labels. On four soft-label benchmarks the approach substantially lowers cross-entropy and Brier score relative to soft-label and ensemble baselines, keeps micro-F1 competitive, and aligns predictions more closely with the full annotator distributions. The result matters for any setting—content moderation, hate speech, sentiment—where a single hard label discards genuine human multiplicity and poorly calibrated uncertainty is costly.

Core claim

EDO demonstrates that treating diversity as a signed, reliability-weighted regularizer inside a single differentiable objective—while simultaneously learning ensemble cardinality via Gumbel-Softmax—produces convex combinations of frozen model predictions that match annotator soft-label distributions far more closely than fixed ensembles or single-model soft supervision, cutting cross-entropy 40–78 percent without sacrificing competitive micro-F1.

What carries the argument

The signed diversity regularizer L_Div^(s) = s · (reliability-weighted pairwise ℓ1 disagreement), with s ∈ {−1, +1} chosen on validation data, jointly optimized with soft F1 and class-weighted CE while ensemble weights w and size K are learned via Gumbel-Softmax relaxation.

Load-bearing premise

A single diversity sign picked once on the development set correctly decides whether observed disagreement is genuine subjectivity worth preserving or structural artifact worth suppressing, and that choice generalizes to test data.

What would settle it

On a held-out subjective dataset whose disagreement structure is independently known (true subjectivity versus imbalance-driven noise), if the validation-chosen sign yields higher cross-entropy and worse F1 than the opposite sign or than the same objective with diversity turned off, the steering claim is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can improve soft-label calibration on imbalanced subjective tasks by adding a cheap, validation-tuned signed diversity term rather than hand-designing ensemble size or architecture.
  • Because optimization occurs entirely in prediction space, any set of frozen probabilistic classifiers can be combined without retraining backbones.
  • Reliability weights emerge automatically from the joint gradients, down-weighting members that contribute unstructured noise.
  • The signed coefficient supplies an explicit, controllable direction along the utility–calibration Pareto frontier under genuine label multiplicity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same signed-diversity mechanism could transfer to other partial-label or multi-hypothesis settings where candidate labels arise from sources other than human annotators.
  • An automatic detector of disagreement type would eliminate the need for manual sign selection and improve robustness under distribution shift.
  • Lightweight adapters or heterogeneous experts could enlarge the hypothesis space while retaining the method’s prediction-space efficiency.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Ensemble Diversity Optimization (EDO), a model-agnostic prediction-space framework for subjective NLP classification under annotator disagreement. It jointly optimizes reliability-aware ensemble weights, effective cardinality (via Gumbel–Softmax relaxation of K), and a multi-objective loss combining soft micro-F1, class-weighted cross-entropy, L2 regularization, and a signed reliability-weighted pairwise L1 diversity term L^(s)_Div (s ∈ {−1,+1}). The sign and loss weights are validation-tuned. On four LeWiDi benchmarks (ArMIS, ConvAbuse, HS-Brexit, MD-Agreement) with frozen BERT/AraBERT backbones, EDO-Random reports large CE reductions (40–78% vs Soft-CE/Soft-MD/Top-5/WEL) and lower Soft Brier scores while keeping competitive micro-F1 and competitive or better MD, with multi-seed stability and ablations on aggregation, weighting, and signed diversity.

Significance. If the empirical gains hold, EDO supplies a practical, lightweight, and model-agnostic way to improve probabilistic calibration and alignment with annotator distributions without requiring annotator metadata or end-to-end backbone fine-tuning. The combination of differentiable cardinality selection, class-weighted soft-label CE, and a signed diversity regularizer that can either preserve or suppress disagreement is a useful operationalization of the bias–variance–diversity decomposition for subjective supervision. Strengths include clear multi-objective formulation (Eqs. 2–6, 10–12), extensive ablations (Tables 2–4, App. D–E), multi-seed stability (Table 6), qualitative examples, and public code. The work is of clear interest to the perspectivist/soft-label and ensemble-calibration communities.

major comments (2)
  1. The central claim that signed diversity enables controlled navigation of the utility–calibration trade-off rests on a single fixed s ∈ {−1,+1} chosen on the development set (Methodology §3.3, Eq. 6; hyperparameter search §4.3). Limitations §6 correctly notes that EDO “does not distinguish principled subjective variation from artifact-driven divergence.” Table 2 and Appendix E show that the preferred sign is dataset-dependent (especially under severe imbalance on HS-Brexit). Without an automatic detector or a sensitivity analysis that reports how often the validation-chosen s is suboptimal on test, the claimed steering mechanism remains an untested modeling assumption rather than a demonstrated capability. A concrete addition—e.g., an oracle upper bound with test-set s, or a simple entropy/imbalance heuristic for choosing s—would make the claim load-bearing rather than contingent.
  2. Table 5 reports large CE/BS gains for EDO-Random, yet the comparison mixes single-model soft-label baselines (Soft-CE, Soft-MD) with ensemble methods (Top-5 Voting, WEL). The paper freezes backbones and optimizes only in prediction space; it is therefore unclear how much of the CE reduction is attributable to the signed diversity + Gumbel-K machinery versus simply having a larger, re-weighted ensemble of already-trained models. An ablation that applies the same reliability weighting and learned K without L^(s)_Div (or with λ_Div = 0) on the identical frozen predictions would isolate the contribution of the signed regularizer that the abstract and introduction emphasize.
minor comments (5)
  1. Theorem 1 (Appendix A) only restates the triangle inequality relating pairwise L1 to barycenter dispersion; the “theoretical note” framing slightly overstates novelty. A shorter statement that L_Div is a monotone proxy for weighted predictive spread would suffice.
  2. Figure 2 Spearman correlations are informative but the caption and surrounding text do not report sample size (number of Optuna trials) or multiple-testing correction; the asterisks for p < 0.05 should be interpreted cautiously.
  3. Notation for soft labels is inconsistent: §3.1 uses ¯y_i while later equations use y_i,c for soft targets; a single convention would improve readability.
  4. Table 1 reports Neg:Pos ratios and annotator ranges; adding the average number of annotators per instance (or total annotation count) would help readers judge the reliability of the soft labels.
  5. The qualitative examples (Appendix G) are helpful; rendering the original Arabic script (or providing a reliable transliteration table) would avoid the ASCII workaround.

Circularity Check

1 steps flagged

No significant circularity: empirical multi-objective framework with validation-tuned signed diversity; self-citation to WEL is present but not load-bearing for the central claims or metrics.

specific steps
  1. self citation load bearing [Section 2 (Related Work) and Section 3 (Methodology intro); also Table 5 baselines]
    "EDO extends Weak Ensemble Learning (WEL) [Huang et al., 2025] through three innovations: (i) a signed, reliability-weighted diversity objective; (ii) class-weighted calibration for imbalanced soft labels; and (iii) differentiable ensemble-structure learning, replacing WEL’s derivative-free optimization. ... WEL [Huang et al., 2025] moves toward more principled ensemble optimization by pairing Random Select and Per-Annotator supervision with objectives for utility and calibration."

    WEL shares all three authors and supplies the preprocessed LeWiDi data plus the direct baseline that EDO claims to improve upon. The citation is not load-bearing for any uniqueness claim or for the definition of L_Div/s/Gumbel-K; the central CE reductions are measured against WEL (and three external methods) on held-out test. Flagged only as minor self-citation, not as a circular derivation step.

full rationale

The paper defines a joint loss (Eq. 2) over soft micro-F1 (Eq. 3), class-weighted CE (Eq. 4), reliability-weighted pairwise L1 diversity (Eq. 5) signed by fixed s (Eq. 6), and L2, then learns weights w and cardinality K via Gumbel-Softmax (Eqs. 10-12). Theorem 1 only bounds L_Div by barycenter dispersion via triangle inequality/convexity; it does not force CE/BS/F1 to equal any fitted quantity. Hyperparameters (including s ∈ {−1,+1} and λ·) are selected on development data via Optuna/NSGA-II and evaluated on held-out test against external baselines (Soft-CE, Soft-MD, Top-5 Voting) plus WEL; this is ordinary ML practice, not a prediction that reduces by construction to the fit. Self-citation to WEL (shared authors, same datasets/preprocessing) appears for comparison and as the method being extended, but the reported CE/BS reductions (Table 5) and ablations (Tables 2-4, Appendix E) are independent empirical measurements, not uniqueness theorems or ansatzes imported from the prior work. No equation equates a claimed first-principles result to its own inputs. The acknowledged limitation on s-selection (Section 6) is a generalization concern, not circularity. Score 1 only for the non-load-bearing self-citation; derivation chain is self-contained.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

Load-bearing content is an empirical multi-objective ensemble method. Free parameters are the Optuna-tuned loss weights, diversity sign, Gumbel schedule, and learning rate. Axioms are standard ML practice (soft surrogates, Gumbel–Softmax, frozen-backbone isolation of annotation uncertainty) plus the modeling choice that a single validation-chosen sign correctly steers preserve-vs-suppress. Invented entities are the EDO objective package and the signed diversity regularizer as used here; independent evidence is only the reported benchmark gains.

free parameters (4)
  • λF1, λCE, λDiv, λReg
    Trade-off weights in the joint objective (Eq. 2); Optuna-tuned on dev sets (search [0,1] or log for Reg).
  • s ∈ {−1,+1}
    Diversity direction (preserve vs suppress); fixed per run after validation selection; central control for the claimed trade-off navigation.
  • T0, γ (Gumbel temperature schedule)
    Initial temperature and decay for ensemble-size relaxation; Optuna-tuned.
  • η (learning rate), Kmax
    Optimizer step size and upper bound on ensemble size; tuned/pre-specified.
axioms (5)
  • domain assumption Soft micro-F1 and class-weighted CE are adequate differentiable surrogates for utility and soft-label calibration under imbalance.
    Used as LF1 and LCE in §3.3 without proof that they preserve the evaluation ranking.
  • standard math Gumbel–Softmax relaxation yields valid end-to-end gradients for discrete ensemble size K.
    Invoked via Jang et al. 2017 in §3.5 / App. B.
  • domain assumption Freezing backbone parameters isolates annotator-driven uncertainty so that prediction-space optimization is sufficient.
    Stated in Introduction and §3; limits representational flexibility (Limitations §6).
  • ad hoc to paper A validation-chosen sign s correctly indicates whether observed disagreement is epistemic subjectivity or structural artifact.
    Core of signed diversity (§3.3); paper admits no automatic distinction (Limitations).
  • standard math Pairwise weighted L1 diversity is monotonically related to ensemble predictive dispersion (Theorem 1).
    App. A; triangle inequality / convexity of ℓ1; standard, not paper-specific physics.
invented entities (2)
  • Ensemble Diversity Optimization (EDO) joint objective no independent evidence
    purpose: Unified differentiable loss over weights, size, soft F1, class-weighted CE, signed diversity, and L2 reg.
    The paper’s main proposed framework; evidence is empirical benchmark gains only.
  • Signed reliability-weighted diversity regularizer L^(s)_Div no independent evidence
    purpose: Bidirectional control of intra-ensemble disagreement (preserve s=−1 or suppress s=+1).
    Defined in §3.3 Eqs. 5–6; direction is a free hyperparameter, not learned from first principles.

pith-pipeline@v1.1.0-grok45 · 30014 in / 3221 out tokens · 38724 ms · 2026-07-10T06:33:29.063694+00:00 · methodology

0 comments
read the original abstract

Subjective NLP tasks often exhibit systematic annotator disagreement, requiring models that represent uncertainty rather than collapse it. We introduce Ensemble Diversity Optimization (EDO), a prediction-space framework that jointly optimizes ensemble weights, effective cardinality, and calibration through a unified differentiable objective. EDO learns ensemble composition and size end-to-end via Gumbel-Softmax relaxation and incorporates a signed diversity regularizer, tuned on validation data, to steer optimization toward either preserving or suppressing disagreement. This regularization prevents ensemble collapse and enables controlled navigation of the utility-calibration trade-off. The framework integrates a soft F1 surrogate, class-weighted cross-entropy to address imbalance, and reliability-weighted diversity to regulate intra-ensemble variability. Experiments on four subjective text-classification benchmarks (ArMIS, ConvAbuse, HS-Brexit, MD-Agreement) show that EDO substantially improves probabilistic calibration, reducing cross-entropy (40-78% depending on baseline) and lowering Brier scores relative to Soft-CE, Soft-MD, Top-5 Voting, and WEL, while maintaining competitive F1 and better alignment with annotator distributions. These results demonstrate that jointly optimizing ensemble structure with a signed diversity regularizer provides an efficient, model-agnostic approach for modeling human subjectivity in supervised learning.

Figures

Figures reproduced from arXiv: 2607.08493 by N. R. Abeynayake, Xia Cui, Ziyi Huang.

Figure 1
Figure 1. Figure 1: Illustration of annotator disagreement. Soft-label supervision preserves annotator label distribu￾tions and provides richer training signals [Uma et al., 2020, Rizzi et al., 2024], but continues to optimize a single pre￾dictive model. This setting aligns with partial-label learning (PLL) [Cour et al., 2011], where each instance is associated with a set of candidate labels. Unlike classical PLL, which assum… view at source ↗
Figure 2
Figure 2. Figure 2: Spearman rank correlations between hyperparameters and development-set metrics (F1, CE, MD, BS). Positive [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

102 extracted references · 102 canonical work pages · 3 internal anchors

  1. [1]

    Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators ' Disagreement

    Leonardelli,Elisa and Menini,Stefano and Palmero Aprosio, Alessio and Guerini, Marco and Tonelli, Sara. Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators ' Disagreement. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021

    work page 2021

  2. [2]

    The ``Problem'' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation

    Plank, Barbara. The ``Problem'' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.731

    work page doi:10.18653/v1/2022.emnlp-main.731 2022

  3. [3]

    C onv A buse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI

    Cercas Curry, Amanda and Abercrombie, Gavin and Rieser, Verena. C onv A buse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021.emnlp-main.587

    work page doi:10.18653/v1/2021.emnlp-main.587 2021

  4. [4]

    A r MIS - The A rabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements

    Almanea, Dina and Poesio, Massimo. A r MIS - The A rabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements. Proceedings of the Thirteenth Language Resources and Evaluation Conference. 2022

    work page 2022

  5. [5]

    Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection

    Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection , author=. arXiv preprint arXiv:2106.15896 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , author=

    Modeling Annotator Perspective and Polarized Opinions to Improve Hate Speech Detection , volume=. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , author=. 2020 , month=. doi:10.1609/hcomp.v8i1.7473 , number=

    work page doi:10.1609/hcomp.v8i1.7473 2020

  7. [7]

    Xia Cui and Amila Gamage and Terry Hanley and Tingting Mu , title=. Proc. Interspeech 2021 , pages=

    work page 2021

  8. [8]

    ACM Transactions on Computer-Human Interaction (TOCHI) , volume=

    Machine learning in mental health: A systematic review of the HCI literature to support the development of effective and implementable ML systems , author=. ACM Transactions on Computer-Human Interaction (TOCHI) , volume=. 2020 , publisher=

    work page 2020

  9. [9]

    Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus

    Schuff, Hendrik and Barnes, Jeremy and Mohme, Julian and Pad \'o , Sebastian and Klinger, Roman. Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus. Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. 2017. doi:10.18653/v1/W17-5203

    work page doi:10.18653/v1/w17-5203 2017

  10. [10]

    Journal of Human resources , volume=

    What are we weighting for? , author=. Journal of Human resources , volume=. 2015 , publisher=

    work page 2015

  11. [11]

    Language Resources and Evaluation , volume=

    Resources and benchmark corpora for hate speech detection: a systematic review , author=. Language Resources and Evaluation , volume=. 2021 , publisher=

    work page 2021

  12. [12]

    Manning , booktitle =

    Jeffrey Pennington and Richard Socher and Christopher D. Manning , booktitle =. GloVe: Global Vectors for Word Representation , year =

    work page

  13. [13]

    Advances in Pre-Training Distributed Word Representations , year =

    Mikolov, Tomas and Grave, Edouard and Bojanowski, Piotr and Puhrsch, Christian and Joulin, Armand , booktitle =. Advances in Pre-Training Distributed Word Representations , year =

    work page

  14. [14]

    Dependency Based Embeddings for Sentence Classification Tasks

    Komninos, Alexandros and Manandhar, Suresh. Dependency Based Embeddings for Sentence Classification Tasks. NAACL-HLT. 2016

    work page 2016

  15. [15]

    Proceedings of the 48th annual meeting of the association for computational linguistics , pages=

    Word representations: a simple and general method for semi-supervised learning , author=. Proceedings of the 48th annual meeting of the association for computational linguistics , pages=

    work page

  16. [16]

    Advances in neural information processing systems , pages=

    Distributed representations of words and phrases and their compositionality , author=. Advances in neural information processing systems , pages=

    work page

  17. [17]

    Flair: An easy-to-use framework for state-of-the-art nlp , year =

    Akbik, Alan and Bergmann, Tanja and Blythe, Duncan and Rasul, Kashif and Schweter, Stefan and Vollgraf, Roland , booktitle =. Flair: An easy-to-use framework for state-of-the-art nlp , year =

    work page

  18. [18]

    A Simple but Tough-to-Beat Baseline for Sentence Embeddings , Year =

    Arora, Sanjeev and Liang, Yingyu and Ma, Tengyu , Booktitle =. A Simple but Tough-to-Beat Baseline for Sentence Embeddings , Year =

    work page

  19. [19]

    S em E val-2023 Task 11: Learning with Disagreements ( L e W i D i)

    Leonardelli, Elisa and Abercrombie, Gavin and Almanea, Dina and Basile, Valerio and Fornaciari, Tommaso and Plank, Barbara and Rieser, Verena and Uma, Alexandra and Poesio, Massimo. S em E val-2023 Task 11: Learning with Disagreements ( L e W i D i). Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). 2023. doi:10.18653/v...

    work page doi:10.18653/v1/2023.semeval-1.314 2023

  20. [20]

    Learning part-of-speech taggers with inter-annotator agreement loss

    Plank, Barbara and Hovy, Dirk and S gaard, Anders. Learning part-of-speech taggers with inter-annotator agreement loss. Proceedings of the 14th Conference of the E uropean Chapter of the Association for Computational Linguistics. 2014. doi:10.3115/v1/E14-1078

    work page doi:10.3115/v1/e14-1078 2014

  21. [21]

    Proceedings of the Twentieth International Conference on International Conference on Machine Learning , pages =

    Geibel, Peter and Wysotzki, Fritz , title =. Proceedings of the Twentieth International Conference on International Conference on Machine Learning , pages =. 2003 , isbn =

    work page 2003

  22. [22]

    , booktitle=

    Clinical Entity Recognition Using Cost-Sensitive Structured Perceptron for NTCIR-10 MedNLP. , booktitle=. 2013 , author=

    work page 2013

  23. [23]

    Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Learning to parse with IAA-weighted loss , author=. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    work page 2015

  24. [24]

    Journal of the Royal Statistical Society

    Fiducial distributions and Bayes' theorem , author=. Journal of the Royal Statistical Society. Series B (Methodological) , pages=. 1958 , publisher=

    work page 1958

  25. [25]

    The Journal of Machine Learning Research , volume=

    ktrain: A low-code library for augmented machine learning , author=. The Journal of Machine Learning Research , volume=. 2022 , publisher=

    work page 2022

  26. [26]

    BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019. doi:10.18653/v...

    work page doi:10.18653/v1/n19-1423 2019

  27. [27]

    ACM Computing Surveys (CSUR) , volume=

    Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions , author=. ACM Computing Surveys (CSUR) , volume=. 2018 , publisher=

    work page 2018

  28. [28]

    and Fornaciari, Tommaso and Hovy, Dirk and Paun, Silviu and Plank, Barbara and Poesio, Massimo , title =

    Uma, Alexandra N. and Fornaciari, Tommaso and Hovy, Dirk and Paun, Silviu and Plank, Barbara and Poesio, Massimo , title =. 2022 , issue_date =. doi:10.1613/jair.1.12752 , journal =

    work page doi:10.1613/jair.1.12752 2022

  29. [29]

    Emotion Representation Mapping for Automatic Lexicon Construction (Mostly) Performs on Human Level

    Buechel, Sven and Hahn, Udo. Emotion Representation Mapping for Automatic Lexicon Construction (Mostly) Performs on Human Level. Proceedings of the 27th International Conference on Computational Linguistics. 2018

    work page 2018

  30. [30]

    xiacui at S em E val-2023 Task 11: Learning a Model in Mixed-Annotator Datasets Using Annotator Ranking Scores as Training Weights

    Cui, Xia. xiacui at S em E val-2023 Task 11: Learning a Model in Mixed-Annotator Datasets Using Annotator Ranking Scores as Training Weights. Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). 2023. doi:10.18653/v1/2023.semeval-1.148

    work page doi:10.18653/v1/2023.semeval-1.148 2023

  31. [31]

    Leveraging Annotator Disagreement for Text Classification

    Xu, Jin and Theune, Mari. Leveraging Annotator Disagreement for Text Classification. Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024). 2024

    work page 2024

  32. [32]

    How do annotators label short texts? Toward understanding the temporal dynamics of tweet labeling , journal =

    Stefan Räbiger and Myra Spiliopoulou and Yücel Saygın , keywords =. How do annotators label short texts? Toward understanding the temporal dynamics of tweet labeling , journal =. 2018 , issn =. doi:https://doi.org/10.1016/j.ins.2018.05.036 , url =

    work page doi:10.1016/j.ins.2018.05.036 2018

  33. [33]

    Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media

    Cook, Owen and Grimshaw, Charlie and Wu, Ben Peng and Dillon, Sophie and Hicks, Jack and Jones, Luke and Smith, Thomas and Szert, Matyas and Song, Xingyi. Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media. Findings of the Association for Computational Linguistics: NAACL 2025. 2025

    work page 2025

  34. [34]

    Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space , url =

    Du, Shangchen and You, Shan and Li, Xiaojie and Wu, Jianlong and Wang, Fei and Qian, Chen and Zhang, Changshui , booktitle =. Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space , url =

    work page

  35. [35]

    Soft metrics for evaluation with disagreements: an assessment

    Rizzi, Giulia and Leonardelli, Elisa and Poesio, Massimo and Uma, Alexandra and Pavlovic, Maja and Paun, Silviu and Rosso, Paolo and Fersini, Elisabetta. Soft metrics for evaluation with disagreements: an assessment. Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024. 2024

    work page 2024

  36. [36]

    Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , author=

    A Case for Soft Loss Functions , volume=. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , author=. 2020 , month=. doi:10.1609/hcomp.v8i1.7478 , abstractNote=

    work page doi:10.1609/hcomp.v8i1.7478 2020

  37. [37]

    and Provost, Foster and Ipeirotis, Panagiotis G

    Sheng, Victor S. and Provost, Foster and Ipeirotis, Panagiotis G. , title =. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =. 2008 , isbn =. doi:10.1145/1401890.1401965 , abstract =

    work page doi:10.1145/1401890.1401965 2008

  38. [38]

    Why Don ' t You Do It Right? Analysing Annotators' Disagreement in Subjective Tasks

    Sandri, Marta and Leonardelli, Elisa and Tonelli, Sara and Jezek, Elisabetta. Why Don ' t You Do It Right? Analysing Annotators' Disagreement in Subjective Tasks. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023. doi:10.18653/v1/2023.eacl-main.178

    work page doi:10.18653/v1/2023.eacl-main.178 2023

  39. [39]

    Transactions of the Association for Computational Linguistics , volume =

    Davani, Aida Mostafazadeh and Díaz, Mark and Prabhakaran, Vinodkumar , title =. Transactions of the Association for Computational Linguistics , volume =. 2022 , month =. doi:10.1162/tacl_a_00449 , url =

    work page doi:10.1162/tacl_a_00449 2022

  40. [40]

    Toward Annotator Group Bias in Crowdsourcing

    Liu, Haochen and Thekinen, Joseph and Mollaoglu, Sinem and Tang, Da and Yang, Ji and Cheng, Youlong and Liu, Hui and Tang, Jiliang. Toward Annotator Group Bias in Crowdsourcing. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. doi:10.18653/v1/2022.acl-long.126

    work page doi:10.18653/v1/2022.acl-long.126 2022

  41. [41]

    Data-Driven or Dataless? Detecting Indicators of Mental Health Difficulties and Negative Life Events in Financial Resilience Using Prompt-Based Learning , year=

    Xia Cui and Terry Hanley and Muj Choudhury and Tingting Mu , booktitle=. Data-Driven or Dataless? Detecting Indicators of Mental Health Difficulties and Negative Life Events in Financial Resilience Using Prompt-Based Learning , year=

    work page

  42. [42]

    Aroyo, Lora and Welty, Chris , title =. AI Mag. , month = mar, pages =. 2015 , issue_date =. doi:10.1609/aimag.v36i1.2564 , abstract =

    work page doi:10.1609/aimag.v36i1.2564 2015

  43. [43]

    Self-Supervised Quality Estimation for Machine Translation

    Zheng, Yuanhang and Tan, Zhixing and Zhang, Meng and Maimaiti, Mieradilijiang and Luan, Huanbo and Sun, Maosong and Liu, Qun and Liu, Yang. Self-Supervised Quality Estimation for Machine Translation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021.emnlp-main.267

    work page doi:10.18653/v1/2021.emnlp-main.267 2021

  44. [44]

    Comparing B ayesian Models of Annotation

    Paun, Silviu and Carpenter, Bob and Chamberlain, Jon and Hovy, Dirk and Kruschwitz, Udo and Poesio, Massimo. Comparing B ayesian Models of Annotation. Transactions of the Association for Computational Linguistics. 2018. doi:10.1162/tacl_a_00040

    work page doi:10.1162/tacl_a_00040 2018

  45. [45]

    A Probabilistic Annotation Model for Crowdsourcing Coreference

    Paun, Silviu and Chamberlain, Jon and Kruschwitz, Udo and Yu, Juntao and Poesio, Massimo. A Probabilistic Annotation Model for Crowdsourcing Coreference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018. doi:10.18653/v1/D18-1218

    work page doi:10.18653/v1/d18-1218 2018

  46. [46]

    Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =

    Xu, Kaidi and Shi, Zhouxing and Zhang, Huan and Wang, Yihan and Chang, Kai-Wei and Huang, Minlie and Kailkhura, Bhavya and Lin, Xue and Hsieh, Cho-Jui , title =. Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =. 2020 , isbn =

    work page 2020

  47. [47]

    Data and Parameter Scaling Laws for Neural Machine Translation

    Gordon, Mitchell A and Duh, Kevin and Kaplan, Jared. Data and Parameter Scaling Laws for Neural Machine Translation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021.emnlp-main.478

    work page doi:10.18653/v1/2021.emnlp-main.478 2021

  48. [48]

    When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks

    Fleisig, Eve and Abebe, Rediet and Klein, Dan. When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.415

    work page doi:10.18653/v1/2023.emnlp-main.415 2023

  49. [49]

    Diverse Instance-Weighting Ensemble Based on Region Drift Disagreement for Concept Drift Adaptation , year=

    Liu, Anjin and Lu, Jie and Zhang, Guangquan , journal=. Diverse Instance-Weighting Ensemble Based on Region Drift Disagreement for Concept Drift Adaptation , year=

    work page

  50. [50]

    Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting

    Zhang, Guanhua and Bai, Bing and Zhang, Junqi and Bai, Kun and Zhu, Conghui and Zhao, Tiejun. Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. doi:10.18653/v1/2020.acl-main.380

    work page doi:10.18653/v1/2020.acl-main.380 2020

  51. [51]

    Neural Processing Letters , author =

    A. Neural Processing Letters , author =. 2024 , pages =. doi:10.1007/s11063-024-11692-z , abstract =

    work page doi:10.1007/s11063-024-11692-z 2024

  52. [52]

    A ra BERT : Transformer-based Model for A rabic Language Understanding

    Antoun, Wissam and Baly, Fady and Hajj, Hazem. A ra BERT : Transformer-based Model for A rabic Language Understanding. 2020

    work page 2020

  53. [53]

    Tune: A Research Platform for Distributed Model Selection and Training

    Tune: A Research Platform for Distributed Model Selection and Training , author=. arXiv preprint arXiv:1807.05118 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  54. [54]

    Proceedings of the 25th

    Optuna: A Next-generation Hyperparameter Optimization Framework , author=. Proceedings of the 25th

    work page

  55. [55]

    Proceedings of the AAAI Conference on Artificial Intelligence , author=

    Toward a Perspectivist Turn in Ground Truthing for Predictive Computing , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2023 , month=. doi:10.1609/aaai.v37i6.25840 , abstractNote=

    work page doi:10.1609/aaai.v37i6.25840 2023

  56. [56]

    Improving Random Forest Method to Detect Hatespeech and Offensive Word , year=

    Nugroho, Kristiawan and Noersasongko, Edy and Purwanto and Muljono and Fanani, Ahmad Zainul and Affandy and Basuki, Ruri Suko , booktitle=. Improving Random Forest Method to Detect Hatespeech and Offensive Word , year=

    work page

  57. [57]

    Hammad and Nisar, Shibli and Ahmad, Awais and Iqbal, Waseem , journal=

    Khan, Anas Ali and Iqbal, M. Hammad and Nisar, Shibli and Ahmad, Awais and Iqbal, Waseem , journal=. Offensive Language Detection for Low Resource Language Using Deep Sequence Model , year=

    work page

  58. [58]

    1999 , publisher=

    CRC standard probability and statistics tables and formulae , author=. 1999 , publisher=

    work page 1999

  59. [59]

    The Advanced Theory of Statistics. Vol. 3. , author=. Biometrics , volume=. 1969 , publisher=

    work page 1969

  60. [60]

    MIND at S em E val-2023 Task 11: From Uncertain Predictions to Subjective Disagreement

    Rizzi, Giulia and Astorino, Alessandro and Scalena, Daniel and Rosso, Paolo and Fersini, Elisabetta. MIND at S em E val-2023 Task 11: From Uncertain Predictions to Subjective Disagreement. Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). 2023. doi:10.18653/v1/2023.semeval-1.77

    work page doi:10.18653/v1/2023.semeval-1.77 2023

  61. [61]

    Journal of Global Optimization , author =

    Differential. Journal of Global Optimization , author =. 1997 , pages =. doi:10.1023/A:1008202821328 , abstract =

    work page doi:10.1023/a:1008202821328 1997

  62. [62]

    Learning Whom to Trust with MACE

    Hovy, Dirk and Berg-Kirkpatrick, Taylor and Vaswani, Ashish and Hovy, Eduard. Learning Whom to Trust with MACE. Proceedings of the 2013 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies. 2013

    work page 2013

  63. [63]

    Advances in neural information processing systems , volume=

    Learning with ensembles: How overfitting can be useful , author=. Advances in neural information processing systems , volume=

    work page

  64. [64]

    Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

    Snow, Rion and O ' Connor, Brendan and Jurafsky, Daniel and Ng, Andrew. Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. 2008

    work page 2008

  65. [65]

    , title =

    Rodrigues, Filipe and Pereira, Francisco C. , title =. 2018 , isbn =

    work page 2018

  66. [66]

    , title =

    Beutel, Alex and Chen, Jilin and Doshi, Tulsee and Qian, Hai and Woodruff, Allison and Luu, Christine and Kreitmann, Pierre and Bischof, Jonathan and Chi, Ed H. , title =. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages =. 2019 , isbn =. doi:10.1145/3306618.3314234 , abstract =

    work page doi:10.1145/3306618.3314234 2019

  67. [67]

    Inherent Disagreements in Human Textual Inferences

    Pavlick, Ellie and Kwiatkowski, Tom. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics. 2019. doi:10.1162/tacl_a_00293

    work page doi:10.1162/tacl_a_00293 2019

  68. [68]

    2017 , booktitle=

    Categorical Reparameterization with Gumbel-Softmax , author=. 2017 , booktitle=

    work page 2017

  69. [69]

    Proceedings of the 5th International Conference on Learning Representations (ICLR) , year =

    The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables , author =. Proceedings of the 5th International Conference on Learning Representations (ICLR) , year =

    work page

  70. [70]

    and Choi, Yejin

    Swayamdipta, Swabha and Schwartz, Roy and Lourie, Nicholas and Wang, Yizhong and Hajishirzi, Hannaneh and Smith, Noah A. and Choi, Yejin. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.746

    work page doi:10.18653/v1/2020.emnlp-main.746 2020

  71. [71]

    A Diversity-Promoting Objective Function for Neural Conversation Models

    Li, Jiwei and Galley, Michel and Brockett, Chris and Gao, Jianfeng and Dolan, Bill. A Diversity-Promoting Objective Function for Neural Conversation Models. Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016. doi:10.18653/v1/N16-1014

    work page doi:10.18653/v1/n16-1014 2016

  72. [72]

    DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation

    Zhang, Yizhe and Sun, Siqi and Galley, Michel and Chen, Yen-Chun and Brockett, Chris and Gao, Xiang and Gao, Jianfeng and Liu, Jingjing and Dolan, Bill. DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 2020....

    work page doi:10.18653/v1/2020.acl-demos.30 2020

  73. [73]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    Wolf, Thomas and Debut, Lysandre and Sanh, Victor and Chaumond, Julien and Delangue, Clement and Moi, Anthony and Cistac, Pierric and Rault, Tim and Louf, Remi and Funtowicz, Morgan and Davison, Joe and Shleifer, Sam and von Platen, Patrick and Ma, Clara and Jernite, Yacine and Plu, Julien and Xu, Canwen and Le Scao, Teven and Gugger, Sylvain and Drame, M...

    work page doi:10.18653/v1/2020.emnlp-demos.6 2020

  74. [74]

    Deep Ensembles: A Loss Landscape Perspective

    Deep ensembles: A loss landscape perspective , author=. arXiv preprint arXiv:1912.02757 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  75. [75]

    DH - FBK at S em E val-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection

    Leonardelli, Elisa and Casula, Camilla. DH - FBK at S em E val-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection. Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). 2023. doi:10.18653/v1/2023.semeval-1.261

    work page doi:10.18653/v1/2023.semeval-1.261 2023

  76. [76]

    and Lee, Wee Sun and Chieu, Hai Leong , title =

    Ye, Nan and Chai, Kian Ming A. and Lee, Wee Sun and Chieu, Hai Leong , title =. Proceedings of the 29th International Conference on Machine Learning , pages =. 2012 , isbn =

    work page 2012

  77. [77]

    2022 , journal=

    sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification , author=. 2022 , journal=

    work page 2022

  78. [78]

    1999 , publisher=

    Nonlinear multiobjective optimization , author=. 1999 , publisher=

    work page 1999

  79. [79]

    Proceedings of the 4th Workshop on Perspectivist Approaches to NLP (NLPerspectives) , month =

    LeWiDi-2025 at NLPerspectives: third Edition of the Learning with Disagreements Shared Task , author=. Proceedings of the 4th Workshop on Perspectivist Approaches to NLP (NLPerspectives) , month =. 2025 , publisher =

    work page 2025

  80. [80]

    Huang, Ziyi and Abeynayake, N. R. and Cui, Xia. Weak Ensemble Learning from Multiple Annotators for Subjective Text Classification. Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP. 2025. doi:10.18653/v1/2025.nlperspectives-1.8

    work page doi:10.18653/v1/2025.nlperspectives-1.8 2025

Showing first 80 references.