Pith. sign in

REVIEW 5 major objections 5 minor 90 references

Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs alone annotate search clarifications unreliably; adding confidence-based selective human review cuts effort 24-45% in simulation, but the evaluation is partly built from the ground truth it predicts.

desk verdict Useful empirical study of LLM limits on fine-grained annotation, but the headline HITL reliability claim is an upper bound because flagged test instances are replaced with ground-truth human labels. read the letter →

arxiv 2507.00543 v1 pith:7GXQB5NS submitted 2025-07-01 cs.IR cs.HC

Leila Tavakoli , Hamed Zamani This is my paper
classification cs.IRcs.HC
keywords annotationllmseffortevaluationfine-grainedhumanmodelsmulti-dimensional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether four large language models (GPT-4o, Claude 3, Cohere Command R, and Mistral 7B) can replace human annotators for search clarification tasks, where annotators rate clarification questions on preference, quality, coverage, diversity, and option order. On the MIMICS-Duo dataset, the models are given the same guidelines as crowdworkers and produce labels in zero-shot and few-shot settings. The authors report that all models agree poorly with human majority labels, especially on fine-grained distinctions, and are sensitive to temperature and prompt changes. The authors then propose a human-in-the-loop workflow. Four models each produce a label and a verbal confidence score; a majority vote gives the ensemble label; the mean confidence and the standard deviation of confidence scores decide whether the label is accepted or sent to a human. Thresholds are tuned on a 10% subset using Pareto optimization, then applied to the remaining 90%. They report 24-45% effort reduction while keeping weighted Cohen's kappa above 0.7. The evaluation has two structural problems. First, flagged test instances are replaced with the pre-existing majority-voted human labels that are also the evaluation target, so agreement on those instances is perfect by construction. Second, the paper treats the standard deviation of confidence scores as inter-model disagreement, but confidence variance does not measure whether models chose different labels. Two models can disagree completely while reporting the same high confidence. Because of these issues, the reported reliability gains are partly built into the design rather than demonstrated by the data.
Extended reading notes

Core claim

The abstract states: 'our findings show that this lightweight intervention significantly improves annotation reliability while reducing human effort by up to 45%' and the contributions section claims 'reducing human annotation effort and cost by 24-45% with no considerable quality degradation'. Concretely, the HITL workflow on MIMICS-Duo achieves weighted Cohen's kappa values around 0.76-0.85 across tasks with 24-45% human effort reduction.

Load-bearing premise

The evaluation assumes that substituting pre-existing majority-voted human labels for every flagged test instance faithfully simulates a real human-in-the-loop process, so that the resulting agreement metrics represent live HITL quality. This enters in Section 4.4, Final Step: 'For any instance flagged for human review, we substitute the aggregated model label with the corresponding pre-existing human label'. If real human reviewers would not always match the MIMICS-Duo majority labels, the reported reliability gains are overstated because flagged instances are perfect by construction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper evaluates four LLMs (GPT-4o, Claude 3, Cohere Command R, and Mistral 7B) as annotators for three fine-grained search clarification tasks from MIMICS-Duo (list-wise preference, pair-wise quality, and pair-wise aspect labelling with coverage, diversity, and option-order sub-tasks). It reports that LLMs are poorly calibrated and sensitive to prompts and temperature, and it proposes a human-in-the-loop (HITL) workflow that flags low-confidence or high-variance instances for human review. The paper claims that this workflow reduces human effort by 24-45% while keeping annotation quality comparable to full human annotation, with weighted Cohen's kappas of 0.757-0.847 across tasks. The evaluation, however, is conducted by substituting the pre-existing majority-voted human labels for every flagged test instance, which makes the reported quality metrics partly circular.

Significance. The strengths of the paper are its systematic comparison of multiple proprietary and open-source LLMs, the public release of code and prompts, the inclusion of a temperature and prompt sensitivity analysis, and the use of a real, multi-dimensional annotation dataset. If the HITL claim were supported, the result would be practically useful for reducing annotation cost in nuanced IR tasks, and the task-level analysis of when human oversight is most needed is a helpful qualitative contribution. However, the central quantitative claim rests on an evaluation design that guarantees perfect agreement on a majority of test instances, so the current evidence does not establish the claimed trade-off. The paper is in scope for ICTIR and the data release is valuable, but the headline result needs substantial re-analysis.

major comments (5)
  1. [§4.4, Final Step; §4.5, Eq. (2)] The headline HITL evaluation is circular. The manuscript states that for any instance flagged for human review, "we substitute the aggregated model label with the corresponding pre-existing human label," and those pre-existing labels are exactly the majority-voted MIMICS-Duo labels used as ground truth in Table 2. Since HER (Eq. 2) is the percentage of instances not flagged, the HITL rows in Table 2 contain the ground-truth label on 55-76% of the evaluation instances (e.g., Task 1 has 45% HER, so 55% of labels are human by construction; Task 2 has 26% HER, so 74% are human by construction). Agreement on those instances is perfect by construction, so the reported weighted kappas of 0.757-0.847 are upper bounds on live HITL quality rather than estimates of it.
  2. [§4.4, Steps 5-6] The same substitution is used during threshold selection, so the Pareto front in Figure 3 and the reported optimal thresholds are also selected under the circular construction. For each candidate threshold pair, the flagged labels on the 10% subset are replaced by the human labels before metrics are computed, and the best configuration is chosen by maximizing weighted kappa while minimizing effort. This biases the selection toward configurations that look good under the substitution rule, and the bias propagates to the full-data results. Thresholds should be selected using quality measured only on the automatically accepted subset, or on a separate validation split where flagged labels are not overwritten.
  3. [§5.1-5.3, Table 2] The paper never reports quality on the automatically accepted subset alone. The only instances for which the LLM ensemble actually determines the output are those not flagged (24-45% of the test portion), and no macro precision, kappa, or MAE is reported for that subset. Without this isolation, the claim of "no considerable quality degradation" is unsupported, because a high overall kappa can be driven entirely by the substituted majority. The authors should report agreement on accepted instances separately and, for the flagged instances, evaluate the routing decision itself, for example by comparing the original LLM label to the human label rather than replacing it.
  4. [§6, Discussion] The acknowledgement that the framework "simulated human-in-the-loop decision-making using majority-voted crowdsourced labels" does not address the optimism of the simulation. Live reviewers are not guaranteed to reproduce the majority label, and even if they did, the evaluation still conflates routing accuracy with label quality. The alternative operating point mentioned in the Discussion (if a lower Kw of 0.718 is acceptable, effort reduction increases to 60%) is derived from the same substitution and therefore inherits the same bias.
  5. [§4.4, Step 1; §5.3] The reported effort reduction does not account for the full cost of the initial human-annotated subset. HER (Eq. 2) is computed only on the remaining 90% of the data, but the workflow requires fully human-annotating a 10% tuning subset before any thresholds can be selected; in a deployment from scratch, that cost must be included in the comparison against full human annotation. The statement in Section 5.3 that "even after factoring in the 10% setup cost, the overall reduction in manual effort remained substantial" is not quantified, and the actual net reduction is lower than 24-45% once the 10% subset is included.
minor comments (5)
  1. [Figure 5] One panel labels the model "Mistral B7" instead of "Mistral 7B"; please correct the typo.
  2. [References] Several references are incomplete; for example, reference [10] lists only an arXiv identifier with no year or venue. Please complete the bibliography.
  3. [Table 2] The table does not indicate which confidence and standard-deviation thresholds correspond to each HITL row; adding threshold values to the table or caption would improve reproducibility.
  4. [§4.4, Step 3] The standard deviation of verbalized confidence scores across models is treated as a proxy for inter-model disagreement, but the models use different confidence scales; this assumption should be validated or at least discussed more explicitly.
  5. [§7, Conclusions] The phrase "considered acceptable based on ICTIR CFP" is informal for a paper text; consider rephrasing the limitation statement.

Circularity Check

2 steps flagged · score 7.0 of 10

HITL reliability is partly constructed: flagged test instances are overwritten with gold labels before computing agreement, so the reported kappas are upper bounds rather than independent predictions.

  1. self definitional [Section 4.4, Final Step; Eq. (2); Table 2 HITL rows]
    "To simulate the HITL process, we leverage the existing human annotations from the MIMICS-Duo dataset, which were determined by majority voting among three crowdworkers. For any instance flagged for human review, we substitute the aggregated model label with the corresponding pre-existing human label—emulating a realistic workflow similar to using human annotators via platforms like AMT."

    The HITL output for every flagged instance is set equal to the majority-voted human label that is also the ground truth for all reported agreement metrics. Since HER is 24-45%, the complement (55-76% of the evaluation set) is copied from gold by construction, so a large fraction of the Table 2 kappas (0.757-0.847) is guaranteed perfect agreement rather than measured prediction. The automatically accepted subset—the only portion where the LLM ensemble actually determines the final label—is never evaluated separately, so the headline claim of 'no considerable quality degradation' is an upper bound under the assumption that live reviewers always reproduce the gold majority label.

  2. fitted input called prediction [Section 4.4, Step 6 and Section 5.1]
    "For each combination of confidence score and standard deviation thresholds, we compute key evaluation metrics—described in the following subsection—based on the updated aggregated labels. The optimal threshold pair is identified using Pareto front optimisation, aiming to maximise annotation quality while minimising human involvement."

    The 'updated aggregated labels' used for threshold selection are the outputs of Step 5, in which flagged labels were overwritten with the pre-existing human gold labels. The Pareto front therefore chooses confidence and standard-deviation thresholds by maximizing a Kw that already contains a perfect-agreement block by construction. This makes the threshold fitting dependent on the same substituted labels that later inflate the full-dataset reliability numbers, so the reported optimal configurations are not independent evidence for the HITL claim.

full rationale

The LLM-only baseline results are a genuine external benchmark, and MIMICS-Duo is an independent public dataset; no load-bearing self-citation chain is present. The circularity is localized to the HITL evaluation. In Section 4.4, every instance flagged for human review receives the pre-existing majority-voted human label as its final output, and the same labels are used as ground truth for weighted Cohen's kappa, MAE, and related metrics. With HER values of 24-45%, roughly 55-76% of the test portion is exact gold by construction, guaranteeing a large block of perfect agreement in the reported kappas. The threshold-tuning step inherits the same construction because it selects thresholds on a 10% subset using these substituted labels. The Discussion's admission that the framework 'simulated human-in-the-loop decision-making using majority-voted crowdsourced labels' is honest, but it does not remove the fact that the central reliability claim reduces in part to its own input labels. A live evaluation would need to report quality on the automatically accepted subset alone or treat human-review labels as noisy, independent measurements; without that, the claimed reliability and effort-reduction trade-off is partially constructed.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The workflow relies on fitted thresholds and several modeling assumptions about confidence calibration and disagreement. No new physical or theoretical entities are introduced.

free parameters (5)
  • confidence threshold (c_threshold) per task = Task1: 90; Task2: 80; Coverage: 75; Diversity: 80; Option Order: 75
    Selected on the 10% subset via Pareto front optimization between weighted Cohen's kappa and human effort, then applied to the remaining 90%.
  • standard deviation threshold (SD_threshold) per task = Task1: 14; Task2: 10; Coverage: 12; Diversity: 12; Option Order: 6
    Selected jointly with the confidence threshold on the 10% subset; the values are data-driven, not derived from theory.
  • temperature = 0
    Chosen because it gave the highest alignment with crowd labels; temperatures 0.5 and 1 were also tested.
  • minimum acceptable weighted Cohen's kappa = 0.7
    Chosen by the authors as the quality floor for the Pareto front constraint.
  • subset size for threshold fitting = 10% of 1,034 pairs
    Selected after testing 5%, 10%, and 15%; 10% was judged sufficient for reliable threshold estimation.
assumptions (4)
  • domain assumption Crowdsourced majority labels in MIMICS-Duo are a valid ground truth for annotation quality.
    All LLM and HITL metrics are computed against these majority labels; if the crowd labels are noisy or biased, every reported agreement measure is affected. Invoked in Section 4.2.
  • domain assumption Substituting pre-existing majority human labels for flagged instances faithfully simulates real human-in-the-loop review.
    Section 4.4 Final Step; this makes the HITL metrics an upper bound because flagged outputs are set equal to the evaluation target.
  • domain assumption Averaging verbalised confidence scores across diverse LLMs yields a meaningful shared confidence scale.
    Section 4.4 Step 3; models have different confidence mechanisms, and the paper does not calibrate confidence across models before averaging.
  • ad hoc to paper Standard deviation of confidence scores across models is a proxy for inter-model disagreement.
    Section 4.4 Steps 3 and 5; SD of confidence does not capture label disagreement when models are confidently wrong in opposite directions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications." pith.science (2026). https://pith.science/paper/7GXQB5NS

@misc{pith2026250700543,
  author       = {Pith},
  title        = {Pith review of: Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GXQB5NS}},
  note         = {Machine review of arXiv:2507.00543}
}
read the original abstract

Despite growing interest in using large language models (LLMs) to automate annotation, their effectiveness in complex, nuanced, and multi-dimensional labelling tasks remains relatively underexplored. This study focuses on annotation for the search clarification task, leveraging a high-quality, multi-dimensional dataset that includes five distinct fine-grained annotation subtasks. Although LLMs have shown impressive capabilities in general settings, our study reveals that even state-of-the-art models struggle to replicate human-level performance in subjective or fine-grained evaluation tasks. Through a systematic assessment, we demonstrate that LLM predictions are often inconsistent, poorly calibrated, and highly sensitive to prompt variations. To address these limitations, we propose a simple yet effective human-in-the-loop (HITL) workflow that uses confidence thresholds and inter-model disagreement to selectively involve human review. Our findings show that this lightweight intervention significantly improves annotation reliability while reducing human effort by up to 45%, offering a relatively scalable and cost-effective yet accurate path forward for deploying LLMs in real-world evaluation settings.

Figures

Figures reproduced from arXiv: 2507.00543 by the authors.

Figure 1
Figure 1. Example clarification pane from Bing [79], illus￾trating multi-choice options for an ambiguous query. • Task 1: List-wise Preference Rating — Models simultaneously evaluate multiple clarification panes2 and rank them to assess their relative preferences, evaluating the model’s ability to handle comparative judgements and contextual reasoning. • Task 2: Pair-wise Quality Labelling — Models evaluate the overall qualit… view at source ↗
Figure 2
Figure 2. HITL workflow, presented for one combination of the confidence score and standard deviation thresholds. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Pareto Front for Kw vs. human effort for all Tasks [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Normalised metrics vs. human effort when implementing HITL in annotating Task 1 on 5%, 10% and 15% subsets. Each [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison LLMs and HITL system predictions (P1,...,P5) against the crowdsourcing workers’ labels (T1,...,T5) across [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 47 canonical work pages

  1. [1]

    Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Dhoyazan Al-Turki, Hansi Hettiarachchi, Mohamed Medhat Gaber, Mo- hammed M Abdelsamea, Shadi Basurra, Sima Iranmanesh, Hadeel Saadany, and Edlira Vakaj. 2024. Human-in-the-Loop Learning with LLMs for Efficient RASE Tagging in Building Compliance Regulations. IEEE Access (2024)

    2024

  4. [4]

    Marwah Alaofi, Paul Thomas, Falk Scholer, and Mark Sanderson. 2024. LLMs can be Fooled into Labelling a Document as Relevant: best café near me; this paper is perfectly relevant. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region. 32–41

    2024

  5. [5]

    Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2020. ConvAI3: Generating Clarifying Questions for Open- Domain Dialogue Systems (ClariQ). arXiv preprint arXiv:2009.11352 (2020)

    arXiv 2020

  6. [6]

    Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval. 475–484

    2019

  7. [7]

    Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2023. Open-source large language models outperform crowd workers and approach ChatGPT in text- annotation tasks. arXiv preprint arXiv:2307.02179 101 (2023)

    arXiv 2023

  8. [8]

    Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohammad- masiha Zahedivafa, Juan D Bermeo, Maria Korobeynikova, and Fabrizio Gilardi

Show all 90 references
  1. [9]

    Anthropic. 2024. Claude 3: A Next-Generation Large Language Model. https: //www.anthropic.com Accessed: 2024-11-25

    2024

  2. [10]

    Julian Ashwin, Aditya Chhabra, and Vijayendra Rao. [n. d.]. Using large language models for qualitative analysis can introduce serious bias. arXiv 2023. arXiv preprint arXiv:2309.17147 ([n. d.])

    2023 arXiv

  3. [11]

    Evan Becker and Stefano Soatto. 2024. Cycles of Thought: Measuring LLM Confidence through Stable Explanations. arXiv e-prints (2024), arXiv–2406

    2024

  4. [12]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  5. [13]

    Zefan Cai, Baobao Chang, and Wenjuan Han. 2023. Humanin-the-Loop through Chain-of-Thought. arXiv preprint arXiv:2306.07932 (2023)

    2023 arXiv

  6. [14]

    Irina Carnat, Giovanni Comandé, Daniele Licari, and Chiara De Nigris. 2024. From human-in-the-loop to LLM-in-the-loop for high quality legal dataset. i-lex 17, 1 (2024), 27–40

    2024

  7. [15]

    Khyathi Raghavi Chandu, Linjie Li, Anas Awadalla, Ximing Lu, Jae Sung Park, Jack Hessel, Lijuan Wang, and Yejin Choi. 2024. Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness. arXiv preprint arXiv:2407.01942 (2024)

    2024 arXiv

  8. [16]

    Cheng-Han Chiang and Hung-Yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 15607– 15631

    2023

  9. [17]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017)

    2017

  10. [18]

    Charles LA Clarke and Laura Dietz. 2024. LLM-based relevance assessment still can’t replace human relevance assessment. arXiv preprint arXiv:2412.17156 (2024)

    2024

  11. [19]

    Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213

    1968

  12. [20]

    Cohere. 2024. Cohere Command R: Retrieval-Optimized Large Language Model. https://cohere.ai Accessed: 2024-11-25

    2024

  13. [21]

    Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation 6, 2 (2002), 182–197

    2002

  14. [22]

    Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a Good Data Annotator?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 11173–11195

    2023

  15. [23]

    Dat Duong and Benjamin D Solomon. 2024. Analysis of large-language model versus human performance for genetics questions. European Journal of Human Genetics 32, 4 (2024), 466–468

    2024

  16. [24]

    Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023. Perspectives on large language models for relevance judgment. In Proceedings of the 2023 ACM SIGI...

    2023

  17. [25]

    Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2024. Who Determines What Is Relevant? Humans or AI? Why Not Both? Commun. ACM 67, 4 (2024), 31–34

    2024

  18. [26]

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554 (2023)

    2023 arXiv

  19. [27]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120, 30 (2023), e2305016120

    2023

  20. [28]

    Yeow Chong Goh, Xin Qing Cai, Walter Theseira, Giovanni Ko, and Khiam Aik Khor. 2020. Evaluating human versus machine learning performance in classify- ing research abstracts. Scientometrics 125 (2020), 1197–1212

    2020

  21. [29]

    Tal Golan, Matthew Siegelman, Nikolaus Kriegeskorte, and Christopher Baldas- sano. 2023. Testing the limits of natural language models for predicting human language judgements. Nature Machine Intelligence 5, 9 (2023), 952–964

    2023

  22. [30]

    Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al . 2023. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854 (2023)

    2023 arXiv

  23. [31]

    Michael Heseltine and Bernhard Clemm von Hohenberg. 2024. Large language models as a substitute for human experts in annotating political text. Research & Politics 11, 1 (2024), 20531680241236239

    2024

  24. [32]

    Jason Holmes, Zhengliang Liu, Lian Zhang, Yuzhen Ding, Terence T Sio, Lisa A McGee, Jonathan B Ashman, Xiang Li, Tianming Liu, Jiajian Shen, et al . 2023. Evaluating large language models on a highly-specialized topic, radiation oncol- ogy physics. Frontiers in Oncology 13 (20...

    2023

  25. [33]

    Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. In Companion proceedings of the ACM web conference 2023 . 294–297

    2023

  26. [34]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  27. [35]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) ...

    2017

  28. [36]

    Yuri Kagolovsky and Jochen R Moehr. 2003. Current status of the evaluation of information retrieval. Journal of medical systems 27 (2003), 409–424

    2003

  29. [37]

    Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang

  30. [38]

    Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2024. Openassistant conversations-democratizing large language model alignment. Advances in Neural Informatio...

    2024

  31. [39]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174

    1977

  32. [40]

    In Pro- ceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations

    MEGAnno+: A Human-LLM Collaborative Annotation System. In Pro- ceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations . 168–176

  33. [41]

    Lin Li, Luo Zhong, Guandong Xu, and Masaru Kitsuregawa. 2012. A feature-free search query classification approach using semantic distance. Expert Systems with Applications 39, 12 (2012), 10739–10748

    2012

  34. [42]

    Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu, and Diyi Yang. 2023. CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation. InProceedings of the 2023 Conference on Empirical Methods in Natural L...

    2023

  35. [43]

    Lingyao Li, Lizhou Fan, Shubham Atreja, and Libby Hemphill. 2024. “HOT” ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media. ACM Transactions on the Web 18, 2 (2024), 1–36

    2024

  36. [44]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2511–2522

    2023

  37. [45]

    Sean MacAvaney and Luca Soldaini. 2023. One-shot labeling for automatic rele- vance estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2230–2235

    2023

  38. [46]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 3214–3252

    2022

  39. [47]

    Etienne Ollion, Rubing Shen, Ana Macanovic, and Arnault Chatelain. 2023. Chat- GPT for Text Annotation? Mind the Hype. SocArXiv preprint (2023)

    2023

  40. [48]

    Kensuke Ono and Akira Morita. 2024. Evaluating large language models: Chatgpt- 4, mistral 8x7b, and google gemini benchmarked against mmlu.Authorea Preprints (2024)

    2024

  41. [49]

    Jay Mohta, Kenan Ak, Yan Xu, and Mingwei Shen. 2023. Are large language models good annotators?. In Proceedings on. PMLR, 38–48. Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications ICTIR ’25, July 18, 2025, Padua, Italy

    2023

  42. [50]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35...

    2022

  43. [51]

    Nicholas Pangakis and Samuel Wolken. 2024. Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI. arXiv preprint arXiv:2409.09467 (2024)

    2024 arXiv

  44. [52]

    Lidiia Ostyakova, Veronika Smilga, Kseniia Petukhova, Maria Molchanova, and Daniel Kornev. 2023. Chatgpt vs. crowdsourcing vs. experts: Annotating open- domain conversations with speech functions. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Disco...

    2023

  45. [53]

    Tejaswini Pedapati, Amit Dhurandhar, Soumya Ghosh, Soham Dan, and Prasanna Sattigeri. 2024. Large Language Model Confidence Estimation via Black-Box Access. arXiv preprint arXiv:2406.04370 (2024)

    2024 arXiv

  46. [54]

    Hossein A Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles LA Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggi- oli. 2024. Llmjudge: Llms for relevance judgments.arXiv preprint arXiv:2408.08896 (2024)

    2024 arXiv

  47. [55]

    Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. Automated anno- tation with generative ai requires validation. arXiv preprint arXiv:2306.00176 (2023)

    2023 arXiv

  48. [56]

    Hamidreza Rouzegar and Masoud Makrehchi. 2024. Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation. In Proceedings of The 18th Linguistic Annotation Workshop (LA W-XVIII). 98–111

    2024

  49. [57]

    Jaromir Savelka and Kevin D Ashley. 2023. The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts. Frontiers in Artificial Intelligence 6 (2023), 1279794

    2023

  50. [58]

    Michael V Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. arXiv preprint arXiv:2304.11085 (2023)

    2023 arXiv

  51. [59]

    Ian Soboroff. 2024. Don’t Use LLMs to Make Relevance Judgments. arXiv preprint arXiv:2409.15133 (2024)

    2024 arXiv

  52. [60]

    Yi-Jyun Sun, Suvodip Dey, Dilek Hakkani-Tur, and Gokhan Tur. 2024. Con- fidence Estimation for LLM-Based Dialogue State Tracking. arXiv preprint arXiv:2409.09629 (2024)

    2024 arXiv

  53. [61]

    Akram Sheriff. 2025. TECHNIQUES FOR DERIVING AN LLM AGENT TRUST SCORE FOR DYNAMICALLY TRIGGERING HUMAN-IN-THE-LOOP (HIL) FEED- BACK IN REALTIME FOR AN LLM AGENTIC WORKFLOW. (2025)

    2025

  54. [62]

    Leila Tavakoli, Johanne R Trippas, Hamed Zamani, Falk Scholer, and Mark Sander- son. 2022. Mimics-duo: Offline & online evaluation of search clarification. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval . 3198–3208

    2022

  55. [63]

    Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. From humans to machines: can chatgpt-like llms effectively replace human annotators in nlp tasks. In Workshop Proceedings of the 17th International AAAI Conference on Web and Social Media

    2023

  56. [64]

    Rikiya Takehi, Ellen M Voorhees, and Tetsuya Sakai. 2024. LLM-Assisted Rel- evance Assessments: When Should We Ask LLMs for Help? arXiv preprint arXiv:2411.06877 (2024)

    2024 arXiv

  57. [65]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine- Tuned with Human Feedback. In Procee...

    2023

  58. [66]

    Petter Törnberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588 (2023)

    2023 arXiv

  59. [67]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023. Large lan- guage models can accurately predict searcher preferences, 2023.URL https://arxiv. org/abs/2309.10621 (2023)

    2023 arXiv

  60. [68]

    Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2024. A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look. arXiv preprint arXiv:2411.08275 (2024)

    2024 arXiv

  61. [69]

    Voorhees

    Ellen M. Voorhees. 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing Management 36, 5 (2000), 697–

    2000

  62. [70]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  63. [71]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926 (2023)

    2023 arXiv

  64. [72]

    Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want To Reduce Labeling Cost? GPT-3 Can Help. InFindings of the Association for Computational Linguistics: EMNLP 2021 . 4195–4205

    2021

  65. [73]

    Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao

  66. [74]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is ChatGPT a Good NLG Evaluator? A Preliminary Study. In Proceedings of the 4th New Frontiers in Summarization Workshop. 1–11

    2023

  67. [75]

    Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing Multiple Choice Science Questions. In Proceedings of the 3rd Workshop on Noisy User- generated Text. 94–106

    2017

  68. [76]

    Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze. 2023. Strength in numbers: Estimating confidence of large language models by prompt agreement. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023). 326–362

    2023

  69. [77]

    Wikipedia contributors. 2025. Cranfield experiments. https://en.wikipedia. org/wiki/Cranfield_experiments. https://en.wikipedia.org/wiki/Cranfield_ experiments [Online; accessed 24-April-2025]

    2025

  70. [78]

    In Proceedings of the CHI Conference on Human Factors in Computing Systems

    Human-LLM collaborative annotation through effective verification of LLM labels. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–21

  71. [79]

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966 (2023)

    2023 arXiv

  72. [80]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  73. [81]

    Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198 (2023)

    2023 arXiv

  74. [82]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 (2022)

    2022 arXiv

  75. [83]

    Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck

  76. [84]

    Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zheng Liu, Ji-Rong Wen, and Zhicheng Dou. 2024. INTERS: unlocking the power of large lan- guage models in search with instruction tuning. arXiv preprint arXiv:2401.06532 (2024)

    2024 arXiv

  77. [85]

    Hamed Zamani, Gord Lueck, Everest Chen, Rodolfo Quispe, Flint Luu, and Nick Craswell. 2020. Mimics: A large-scale data collection for search clarification. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 3189–3196

    2020

  78. [89]

    Yiming Zhu, Peixian Zhang, EU Haq, Pan Hui, and Gareth Tyson. 2023. Can ChatGPT reproduce human-generated labels. A Study of Social Computing Tasks (2023), 92–101

    2023

  79. [716]

    https://doi.org/10.1016/S0306-4573(00)00010-8

    doi

  80. [2020]

    In Proceedings of The Web Conference 2020

    Generating clarifying questions for information retrieval. In Proceedings of The Web Conference 2020. 418–428

    2020

  81. [2024]

    Can We Use Large Language Models to Fill Relevance Judgment Holes? arXiv preprint arXiv:2405.05600 (2024)

    2024 arXiv

  82. [2025]

    Journal of Computational Social Science 8, 1 (2025), 1–25

    Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning. Journal of Computational Social Science 8, 1 (2025), 1–25

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.