Pith. sign in

REVIEW 3 major objections 4 minor 65 references

Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Direct training beats inference for spotting subjective arguments

desk verdict Direct subjectivity training beats inference on the ValueEval subset, but the four-annotator target leaves the general claim only partly supported. read the letter →

arxiv 2509.06704 v1 pith:6NB6JO3P submitted 2025-09-08 cs.CL

Amir Homayounirad , Enrico Liscio , Tong Wang , Catholijn M. Jonker , Luciano C.Siebert This is my paper · ORCID
classification cs.CL
keywords subjectivityidentificationannotatordisagreementhumanvaluesvalue-ladenargumentsmulti-annotatorNLPcontrastivelearningargumentationlabelvariation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that, for arguments carrying human values, the best way to predict whether annotators will disagree is to train a model directly on disagreement labels, not to predict each annotator's value choices and then compare them. On the Touché23-ValueEval dataset, direct subjectivity models reach F1 0.80, while inference-based models stay at 0.36 and random guessing at 0.51. Adding contrastive losses does not improve F1 but produces embeddings that separate subjective from non-subjective arguments more cleanly and reduces the model's tendency to do better only on less subjective values. If the result holds, annotation pipelines and deliberation tools can flag arguments likely to be interpreted differently, prompting extra annotations or moderator questions.

What carries the argument

The central object is the binary subjectivity classifier trained directly on disagreement labels, one per value (DS-simple), with optional auxiliary contrastive losses—triplet loss (supervised) and contrastive tension (unsupervised)—added to binary cross-entropy. The direct label is what carries the argument: it bypasses the multi-label value-prediction task entirely, and the comparison with inference-based methods isolates the effect of the learning objective rather than the architecture.

What would settle it

Take the same arguments and collect value annotations from a fresh set of, say, twenty annotators with diverse demographics; if the DS-simple model's subjectivity scores do not predict which items the new group disagrees on, the method has captured the original four annotators' quirks rather than disagreement itself.

Watch

Extended reading notes

Core claim

On the Touché23-ValueEval dataset, the paper defines an argument-value pair as subjective when at least one of four annotators assigns a different set of values for that pair. It compares three inference-based methods—dedicated per-annotator models, shared embeddings with per-annotator heads, and a single model with annotator IDs—against three direct methods that train binary classifiers on the disagreement labels, two of which add contrastive objectives. The direct methods score far higher (best F1 0.80 with DS-simple on Llama versus 0.36 for the best inference approach and 0.51 random), showing that disagreement is better learned as a label in its own right than reconstructed from predicte

Load-bearing premise

The entire evaluation rests on defining 'subjective' as disagreement among four specific annotators chosen for high overlap; if those four are too similar to each other, or if their disagreements are mostly noise, the models are learning how that particular group annotates, not a general property of the arguments.

Editorial extensions

If this is right

  • A model trained to predict disagreement directly can flag arguments that would benefit from additional annotations before they are sent to more annotators.
  • Annotation projects can keep individual value predictions for interpretability while using a separate direct subjectivity head for accuracy, since the two objectives are not interchangeable.
  • The supervised contrastive variant produces embeddings that separate subjective from non-subjective arguments more cleanly, which can support active learning sampling near the decision boundary.
  • Scaling the model used for direct subjectivity identification (BERT to Llama) improves recall substantially, while the same scaling does not rescue inference-based methods—suggesting the inference bottleneck is structural rather than a matter of model size.
  • The weaker dependence of DS-sup on per-label subjectivity means combined losses yield more consistent performance across values with different disagreement rates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's comparison does not isolate whether the gain comes from the direct label or from the paraphrasing data augmentation used only for DS; a testable follow-up would train DS without augmentation to separate these effects.
  • The strong negative correlation for DS-simple (Llama) suggests the model may excel on values where disagreement is rare by exploiting distributional cues; probing with counterfactual rewrites would test whether it learns generic subjectivity or value-specific surface patterns.
  • The approach could transfer to other subjective annotation tasks, such as hate speech or emotion detection, where per-item disagreement labels exist; the paper demonstrates it only for value-laden arguments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses subjectivity in recognizing human values behind arguments. It formalizes an argument as subjective for a value when at least one annotator assigns a different set of values than the others. Two families of methods are proposed: inferred subjectivity identification (IS), which trains per-annotator value prediction models and derives disagreement from predicted value differences, and direct subjectivity identification (DS), which trains binary classifiers per value to predict disagreement directly. The methods are evaluated on a subset of Touché23-ValueEval consisting of eight level-2 values annotated by four selected annotators, using BERT and Llama models. The best result is DS-simple (Llama) at F1 0.80, well above the random baseline of 0.51, while all IS methods are near or below the random baseline. The paper also investigates supervised and unsupervised contrastive losses, finding that they do not improve F1 but affect embedding structure and the correlation with per-value subjectivity.

Significance. If the central claim holds, the work is a useful contribution to the label-variation literature: it provides a concrete comparison between direct subjectivity classification and inference from per-annotator value prediction, and it identifies a large practical gap between the two. Strengths include the public code release, a fixed test set, standard deviations reported for most BERT experiments, a random baseline, and a systematic comparison of several architectural variants. However, the external validity of the main claim—that the method predicts whether annotators will disagree—is not yet established, because the subjectivity labels are derived from only four annotators selected for high overlap, with no held-out annotator validation. The internal DS-versus-IS comparison is coherent, but the title and abstract promise more than the current experimental design can support.

major comments (3)
  1. [§4.2, App. A.1, §2.2] The DS labels are built from exactly four annotators selected for high annotation overlap. Section 2.2 states the method is “not being reliant on specific individuals or groups,” but the binary subjectivity targets are tied to this particular group. No held-out annotator set is used: a model trained on these four annotators may learn the disagreement pattern of a small, similar group rather than generalizable subjectivity. The paper itself concedes in Section 7 that annotation disagreement can be confounded with noise. I recommend annotator-level cross-validation (e.g., train on three annotators, evaluate on the fourth; or resample annotator subsets) and a report of how stable DS performance is across annotator configurations. In addition, Appendix A.1 says the selection was “the first four annotators who annotated the most,” while Section 4.2 says “highest annotation overlap”; these are
  2. [§5, Table 1, App. B] The abstract and Section 5.1 state that direct subjectivity identification “significantly improves” performance, but no statistical significance test is reported. Appendix B provides means and standard deviations over five runs for most BERT methods, yet there is no paired test (e.g., paired bootstrap, McNemar) or confidence intervals for the comparisons that drive the conclusions (DS vs. IS, DS-simple vs. DS-sup). Moreover, the Llama results, which include the best overall system, are reported without standard deviations. Since the central claim is comparative, the authors should add explicit significance tests or at least report per-seed variation for the Llama experiments.
  3. [§3.1, §7] The operational definition of subjectivity as “at least one annotator assigns a different set of values compared to others” is very sensitive with only four annotators: a single divergent annotation makes the argument subjective. The paper acknowledges in Section 7 that disagreement can be confounded with noise, but it does not attempt to distinguish systematic disagreement from isolated error. This is load-bearing because the DS models are trained to reproduce this binary signal. A robustness analysis varying the disagreement threshold (e.g., requiring two or more annotators to differ) would show whether the reported gains depend on the current dichotomization.
minor comments (4)
  1. [§4.2 vs. App. A.1] The main text reports 9,324 natural language arguments, while Appendix A.1 reports 5,270. Please reconcile the dataset size.
  2. [§8] The Limitations section contains a garbled sentence: “The first two methods from ISV including —ISV-each, ISV-shared, and DS-simple are computationally expensive,” and the names ISV-each/ISV-shared do not match the method names introduced in Section 3. Please correct.
  3. [App. B] The appendix tables use labels DBCV, SBDCV, SBCV, DBV, SBV without mapping them to the method names in Section 3. A legend or renaming would help readers.
  4. [Figure 4] The t-SNE visualizations are illustrative but the claims about embedding separation and grouping would be stronger with a quantitative metric (e.g., silhouette score or nearest-neighbor agreement).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: standard supervised evaluation with externally sourced annotations and a held-out test set.

full rationale

The paper's central claim—that direct subjectivity identification (DS) outperforms inference from value prediction (IS)—rests on a standard supervised learning comparison. Subjectivity labels are derived from the Touché23-ValueEval crowd annotations (Section 4.2), and DS models are trained with binary cross-entropy and optional contrastive losses to predict those labels; IS models are trained to predict per-annotator value labels, and the same operational definition of disagreement is applied to their outputs. Evaluation uses a fixed held-out test set with hyperparameters chosen on validation, and Appendix B reports standard deviations over five runs with different train/validation splits. No fitted parameter is renamed as a prediction, no equation reduces the target to the model's own outputs by construction, and no load-bearing premise is justified only by a self-citation. The selection of four high-overlap annotators and the possible confound between subjectivity and annotation noise are real external-validity limitations, explicitly acknowledged in Section 7 and the Limitations section, but these concern construct validity rather than circularity: the model's target is defined by annotator disagreement, not by the model's predictions. The comparison between DS and IS is therefore self-contained and empirically meaningful.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. It relies on a specific definition of subjectivity, a small selected subset of annotators and values, and standard transformer assumptions. The main free parameters are the contrastive loss weights and margin, plus data augmentation settings, all chosen by hand on validation. These choices are not fitted to the test set, so they do not inflate circularity, but they do affect how well the results generalize.

free parameters (4)
  • contrastive_loss_weight_alpha = 1.0 (DS-sup), 5.0 (DS-unsup)
    Weight on the auxiliary contrastive loss in the combined loss L = L_BCE + alpha * L_CL; chosen by hand, affects final embeddings and performance.
  • triplet_margin_m = 1.0
    Margin in the triplet loss for DS-sup; enforces minimum separation between positive and negative pairs, set by hand.
  • contrastive_temperature_tau = not reported
    Temperature in the contrastive tension loss for DS-unsup; not listed in the hyperparameter table, likely left at a default value.
  • paraphrase_generation_temperature = 2.0
    Used for data augmentation of the minority class in DS methods (Appendix A.2); impacts the distribution of synthetic samples used in training.
assumptions (4)
  • ad hoc to paper An argument is subjective if at least one annotator assigns a different set of values than the others (Section 3.1).
    This binary operationalization is arbitrary; other thresholds (e.g., majority disagreement, variance-based) would change the labels and potentially the results.
  • domain assumption Disagreement among annotators reflects subjectivity and not annotation noise (Section 3.1, acknowledged as a confound in Section 7).
    The paper itself acknowledges that differences in annotation can be confounded with noise. If noise dominates, the trained models are learning noise, not subjectivity.
  • ad hoc to paper The four annotators with highest annotation overlap and the eight most frequently annotated values are representative enough to evaluate subjectivity.
    This selection reduces the dataset and biases toward annotators who agree more, likely understating true subjectivity and limiting generalizability.
  • domain assumption Standard transformer embeddings (BERT, Llama) capture the semantic information needed for subjectivity detection.
    The methods rely on fine-tuned language models without any external knowledge of annotator backgrounds or deliberation context.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments." pith.science (2026). https://pith.science/paper/6NB6JO3P

@misc{pith2026250906704,
  author       = {Pith},
  title        = {Pith review of: Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NB6JO3P}},
  note         = {Machine review of arXiv:2509.06704}
}
read the original abstract

Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.

Figures

Figures reproduced from arXiv: 2509.06704 by the authors.

Figure 1
Figure 1. Representation of how the method processes input text to predict values for individual annotators and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Representation of how the method processes input text to identify subjectivity for each value label [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Counts and ratio of subjective (S, red) and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of 2D sentence embeddings visu [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 38 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sai Vamsi Alisetti. 2021. Paraphrase generator with t5

    work page 2021

  4. [4]

    Milad Alshomary, Roxanne El Baff, Timon Gurcke, and Henning Wachsmuth. 2022. https://doi.org/10.18653/v1/2022.acl-long.601 The moral debater: A study on the computational generation of morally framed arguments . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8782--8797, Dublin, Ire...

    work page doi:10.18653/v1/2022.acl-long.601 2022

  5. [5]

    Oscar Araque, Lorenzo Gatti, and Kyriaki Kalimeri. 2020. Moralstrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction. Knowledge-based systems, 191:105184

    work page 2020

  6. [6]

    Taylor, Mark D \'i az, Christopher Michael Homan, Alicia Parrish, Greg Serapio-Garc \'i a, Vinodkumar Prabhakaran, and Ding Wang

    Lora Aroyo, Alex S. Taylor, Mark D \'i az, Christopher Michael Homan, Alicia Parrish, Greg Serapio-Garc \'i a, Vinodkumar Prabhakaran, and Ding Wang. 2023. https://api.semanticscholar.org/CorpusID:259203842 Dices dataset: Diversity in conversational ai evaluation for safety . ArXiv, abs/2306.11247

    arXiv 2023

  7. [7]

    Connor Baumler, Anna Sotnikova, and Hal Daum \'e III. 2023. https://doi.org/10.18653/v1/2023.findings-acl.658 Which examples should be multiply annotated? active learning when annotators may disagree . In Findings of the Association for Computational Linguistics: ACL 2023, pages 10352--10371, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-acl.658 2023

  8. [8]

    Trevor J. M. Bench-Capon. 2003. https://doi.org/10.1093/logcom/13.3.429 Persuasion in Practical Argument Using Value-based Argumentation Frameworks . Journal of Logic and Computation, 13(3):429--448

    doi:10.1093/logcom/13.3.429 2003

Show all 65 references
  1. [9]

    Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Ar \'e chiga, and Tengyu Ma. 2019. https://api.semanticscholar.org/CorpusID:189998981 Learning imbalanced datasets with label-distribution-aware margin loss . In Neural Information Processing Systems

    2019

  2. [10]

    Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylip \"a \"a Hellqvist, and Magnus Sahlgren. 2021. https://api.semanticscholar.org/CorpusID:235613354 Semantic re-tuning with contrastive tension . In International Conference on Learning Representations

    2021

  3. [11]

    J. D. Chandler and S. L. Vargo. 2011. https://doi.org/10.1177/1470593110393713 Contextualization and value-in-context: how context frames exchange . Marketing Theory, 11:35--49

    2011 doi

  4. [12]

    Naihao Deng, Xinliang Zhang, Siyang Liu, Winston Wu, Lu Wang, and Rada Mihalcea. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.832 You are what you annotate: Towards better models through annotator representations . In Findings of the Association for Computational Ling...

    2023 doi

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://api.semanticscholar.org/CorpusID:52967399 Bert: Pre-training of deep bidirectional transformers for language understanding . In North American Chapter of the Association for Computational Linguistics

    2019

  6. [14]

    Edvardsson, B

    B. Edvardsson, B. Tronvoll, and T. Gruber. 2010. https://doi.org/10.1007/s11747-010-0200-y Expanding understanding of service exchange and value co-creation: a social construction approach . Journal of the Academy of Marketing Science, 39:327--339

    2010 doi

  7. [15]

    Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.415 When the majority is wrong: Modeling annotator disagreement for subjective tasks . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6...

    2023 doi

  8. [16]

    Joseph L. Fleiss. 1971. https://api.semanticscholar.org/CorpusID:143544759 Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378--382

    1971

  9. [17]

    Nitesh Goyal, Ian Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. https://arxiv.org/abs/2205.00501 Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation . Preprint, arXiv:2205.00501

    2022 arXiv

  10. [18]

    Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Advances in experimental social psychology, volume 47, pages 55--130. Elsevier

    2013

  11. [19]

    Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020. https://api.semanticscholar.org/CorpusID:226237047 Supervised contrastive learning for pre-trained language model fine-tuning . ArXiv, abs/2011.01403

    2020 arXiv

  12. [20]

    Amy Gutmann and Dennis F Thompson. 2004. Why deliberative democracy? Princeton University Press

    2004

  13. [21]

    Hadsell, S

    R. Hadsell, S. Chopra, and Y. LeCun. 2006. https://doi.org/10.1109/CVPR.2006.100 Dimensionality reduction by learning an invariant mapping . In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2, pages 1735--1742

    2006 doi

  14. [22]

    Jonathan Haidt and Jesse Graham. 2007. When morality opposes justice: Conservatives have moral intuitions that liberals may not recognize. Social justice research, 20(1):98--116

    2007

  15. [23]

    Joe Hoover, Gwenyth Portillo-Wightman, Leigh Yeh, Shreya Havaldar, Aida Mostafazadeh Davani, Ying Lin, Brendan Kennedy, Mohammad Atari, Zahra Kamel, Madelyn Mendlen, et al. 2020. Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment. Social...

    2020

  16. [24]

    Horbel, B

    C. Horbel, B. Popp, H. Woratschek, and B. Wilson. 2016. https://doi.org/10.1080/02642069.2016.1255730 How context shapes value co-creation: spectator experience of sport events . The Service Industries Journal, 36:510--531

    2016

  17. [25]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  18. [26]

    Wormley, and Adam H

    Xiaolei Huang, Alexandra S. Wormley, and Adam H. Cohen. 2022. https://api.semanticscholar.org/CorpusID:248227355 Learning to adapt domain shifts of moral values via instance weighting . Proceedings of the 33rd ACM Conference on Hypertext and Social Media

    2022

  19. [27]

    John Kekes. 1996. The morality of pluralism

    1996

  20. [28]

    Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. 2022. Identifying the human values behind arguments. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4459--4471

    2022

  21. [29]

    Johannes Kiesel, Milad Alshomary, Nailia Mirzakhmedova, Maximilian Heinrich, Nicolas Handke, Henning Wachsmuth, and Benno Stein. 2023. https://doi.org/10.18653/v1/2023.semeval-1.313 S em E val-2023 task 4: V alue E val: Identification of human values behind arguments . In Proc...

    2023 doi

  22. [30]

    Johannes Kiesel et al. 2024. https://doi.org/10.1007/978-3-031-71908-0_14 Overview of Touch \'e 2024: Argumentation Systems . In Experimental IR Meets Multilinguality, Multimodality, and Interaction. 15th International Conference of the CLEF Association (CLEF 2024), Lecture No...

    2024 doi

  23. [31]

    Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. https://doi.org/10.18653/v1/2021.acl-long.197 Self-guided contrastive learning for BERT sentence representations . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati...

    2021 doi

  24. [32]

    Milkowski, and Przemyslaw Kazienko

    Jan Kocoń, Marcin Gruza, Julita Bielaniewicz, Damian Grimling, Kamil Kanclerz, P. Milkowski, and Przemyslaw Kazienko. 2021. https://api.semanticscholar.org/CorpusID:246288645 Learning personal human biases and representations for subjective tasks in natural language processing...

    2021

  25. [33]

    Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 30(3):411--433

    2004

  26. [34]

    Yuanyuan Lei, Md Messal Monem Miah, Ayesha Qamar, Sai Ramana Reddy, Jonathan Tong, Haotian Xu, and Ruihong Huang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.293 EMONA : Event-level moral opinions in news articles . In Proceedings of the 2024 Conference of the North Amer...

    2024 doi

  27. [35]

    Dondera, Andrei Gead a u, Catholijn M

    Enrico Liscio, Alin E. Dondera, Andrei Gead a u, Catholijn M. Jonker, and Pradeep K. Murukannaiah. 2022. https://aclanthology.org/2022.findings-naacl.209/ Cross-domain classification of moral values . In Findings of the Association for Computational Linguistics: NAACL 2022, pa...

    2022

  28. [36]

    Enrico Liscio, Luciano C Siebert, Catholijn M Jonker, and Pradeep K Murukannaiah. 2025. https://www.jair.org/index.php/jair/article/view/14958 Value preferences estimation and disambiguation in hybrid participatory systems . Journal of Artificial Intelligence Research, 82:819--850

    2025

  29. [37]

    Siebert, Catholijn M

    Enrico Liscio, Michiel van der Meer, Luciano C. Siebert, Catholijn M. Jonker, Niek Mouter, and Pradeep K. Murukannaiah. 2021. https://www.ifaamas.org/Proceedings/aamas2021/pdfs/p799.pdf Axies: Identifying and Evaluating Context-Specific Values . In Proceedings of the 20th Inte...

    2021

  30. [38]

    Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. 2016. https://api.semanticscholar.org/CorpusID:1829423 Large-margin softmax loss for convolutional neural networks . In International Conference on Machine Learning

    2016

  31. [39]

    Charles G Lord, Lee Ross, and Mark R Lepper. 1979. Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of personality and social psychology, 37(11):2098

    1979

  32. [40]

    John L Mackie. 1988. The subjectivity of values. Essays on moral realism, pages 95--118

    1988

  33. [41]

    Nailia Mirzakhmedova, Johannes Kiesel, Milad Alshomary, Maximilian Heinrich, Nicolas Handke, Xiaoni Cai, Barriere Valentin, Doratossadat Dastgheib, Omid Ghahroodi, Mohammad Ali Sadraei, Ehsaneddin Asgari, Lea Kawaletz, Henning Wachsmuth, and Benno Stein. 2023. https://api.sema...

    2023

  34. [42]

    Aida Mostafazadeh Davani, Mark Diaz, Dylan K Baker, and Vinodkumar Prabhakaran. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1029 D 3 CODE : Disentangling disagreements in data across cultures on offensiveness detection and evaluation . In Proceedings of the 2024 Conferen...

    2024 doi

  35. [43]

    Aida Mostafazadeh Davani, Mark D \' az, and Vinodkumar Prabhakaran. 2022. https://doi.org/10.1162/tacl_a_00449 Dealing with disagreements: Looking beyond the majority vote in subjective annotations . Transactions of the Association for Computational Linguistics, 10:92--110

    2022 doi

  36. [44]

    Matthias Orlikowski, Paul R \"o ttger, Philipp Cimiano, and Dirk Hovy. 2023. https://doi.org/10.18653/v1/2023.acl-short.88 The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics . In Proceedings of the 61st Annual Meeting of the Asso...

    2023 doi

  37. [45]

    Jeongwoo Park, Enrico Liscio, and Pradeep Murukannaiah. 2024. https://aclanthology.org/2024.findings-eacl.45 Morality is non-binary: Building a pluralist moral sentence embedding space using contrastive learning . In Findings of the Association for Computational Linguistics: E...

    2024

  38. [46]

    Frey, Michael B

    Desmond Upton Patton, Philipp Blandfort, William R. Frey, Michael B. Gaskell, and Svebor Karaman. 2019. https://api.semanticscholar.org/CorpusID:102352155 Annotating social media data from vulnerable populations: Evaluating disagreement between domain experts and graduate stud...

    2019

  39. [47]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The `` problem '' of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Ab...

    2022 doi

  40. [48]

    John Rawls. 2002. John Rawls: Political liberalism and the law of peoples, volume 2. Taylor & Francis

    2002

  41. [49]

    Lee Ross and Andrew Ward. 2013. Naive realism in everyday life: Implications for social conflict and misunderstanding. In Values and knowledge, pages 103--135. Psychology Press

    2013

  42. [50]

    Pierrehumbert

    Paul R \"o ttger, Bertie Vidgen, Dirk Hovy, and Janet B. Pierrehumbert. 2021. https://api.semanticscholar.org/CorpusID:245130931 Two contrasting data annotation paradigms for subjective nlp tasks . In North American Chapter of the Association for Computational Linguistics

    2021

  43. [51]

    Mike Schaekermann, Joslin Goh, Kate Larson, and Edith Law. 2018. https://doi.org/10.1145/3274423 Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work . Proc. ACM Hum.-Comput. Interact., 2(CSCW)

    2018 doi

  44. [52]

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. https://doi.org/10.1109/cvpr.2015.7298682 Facenet: A unified embedding for face recognition and clustering . In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 815–823. IEEE

    2015

  45. [53]

    Schwartz

    Shalom H. Schwartz. 1994. https://doi.org/10.1111/j.1540-4560.1994.tb01196.x Are there universal aspects in the structure and contents of human values? Journal of Social Issues, 50(4):19--45

    1994

  46. [54]

    Shalom H. Schwartz, Jan Cieciuch, Michele Vecchione, Eldad Davidov, Ronald Fischer, Constanze Beierlein, Alice Ramos, Markku Verkasalo, Jan-Erik L \"o nnqvist, Kursad Demirutku, Ozlem Dirilen-Gumus, and Mark Konty. 2012. https://api.semanticscholar.org/CorpusID:20456630 Refini...

    2012

  47. [55]

    Rithik Appachi Senthilkumar, Amir Homayounirad, and Luciano Cavalcante Siebert. 2024. Leveraging large language models to identify the values behind arguments. In International Workshop on Value Engineering in AI, pages 87--103. Springer

    2024

  48. [56]

    Barry Stroud. 1988. The study of human nature and the subjectivity of value. The Tanner Lectures on Human Value

    1988

  49. [57]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  50. [58]

    T5 paraphrase paws

    Vamsi. T5 paraphrase paws. https://huggingface.co/Vamsi/T5_Paraphrase_Paws. Accessed: 2024-12-06

    2024

  51. [59]

    Murukannaiah, and Enrico Liscio

    Michiel van der Meer, Neele Falk, Pradeep K. Murukannaiah, and Enrico Liscio. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1031 Annotator-centric active learning for subjective NLP tasks . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proc...

    2024 doi

  52. [60]

    S. L. Vargo and R. F. Lusch. 2015. https://doi.org/10.1007/s11747-015-0456-3 Institutions and axioms: an extension and update of service-dominant logic . Journal of the Academy of Marketing Science, 44:5--23

    2015 doi

  53. [61]

    Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. 2023. https://doi.org/10.1609/aaai.v37i12.26698 Everyone's voice matters: quantifying annotation disagreement using demographic information . In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty...

    2023 doi

  54. [62]

    Xinpeng Wang and Barbara Plank. 2023. https://arxiv.org/abs/2310.14979 Actor: Active learning with annotator-specific classification heads to embrace human label variation . Preprint, arXiv:2310.14979

    2023 arXiv

  55. [63]

    Zeerak Waseem. 2016. https://doi.org/10.18653/v1/W16-5618 Are you a racist or am I seeing things? annotator influence on hate speech detection on T witter . In Proceedings of the First Workshop on NLP and Computational Social Science , pages 138--142, Austin, Texas. Associatio...

    2016 doi

  56. [64]

    Leon Weber-Genzel, Siyao Peng, Marie-Catherine De Marneffe, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.acl-long.123 V ari E rr NLI : Separating annotation error from human label variation . In Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024 doi

  57. [65]

    Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christopher Homan. 2023. https://doi.org/10.18653/v1/2023.findings-acl.287 Disagreement matters: Preserving label diversity by jointly modeling item and annotator label distributions with...

    2023 doi

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.