Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Co-DETECT is a mixed-initiative annotation framework that uses LLM-flagged low-confidence examples to produce high-level edge-case rules, and reports F1 gains on hate-speech and emotion classification after one iteration.

desk verdict A useful mixed-initiative annotation tool that is currently oversold: the F1 gain in Table 1 is in-sample, so the central claim of generalization is not yet established. read the letter →

arxiv 2507.05010 v1 pith:7LX4QRRV submitted 2025-07-07 cs.CL

Chenfei Xiong , Jingwei Ni , Yu Fan , Vilém Zouhar , Donya Rooein , Lorena Calvo-Bartolomé , Alexander Hoyle , Zhijing Jin
show 5 more authors
Mrinmaya Sachan Markus Leippold Dirk Hovy Mennatallah El-Assady Elliott Ash
This is my paper
classification cs.CL
keywords textclassificationedgecasediscoverycodebookdevelopmentLLMannotationhuman-AIcollaborationmixed-initiativereasoninguserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Co-DETECT is a system that lets a domain expert start from a rough codebook and a few hundred texts, then uses an LLM to annotate the texts, report its own confidence, and describe any example it found ambiguous. A reasoning LLM groups those pointwise descriptions into high-level, generalizable edge-case rules, which the expert can inspect, edit, and append to the codebook before re-annotating. The paper claims this loop improves codebook-driven classification: in one iteration, F1 rose from 0.2144 to 0.2523 on GabHateCorpus hate-speech data, from 0.0300 to 0.3297 on GoEmotions-positive emotion detection, and from 0.2823 to 0.3046 on GoEmotions-negative emotion detection. The motivation is that codebook development, not annotation volume, is often the bottleneck in expert text analysis, and subjective tasks are hard for both human experts and LLMs to codify alone.

What carries the argument

The load-bearing mechanism is a three-step backend that converts individual low-confidence predictions into codebook-ready rules. First, a non-reasoning LLM annotates each text and, when confidence is low, emits an explanation framed as 'when [case], do [action]'. Second, these pointwise descriptions are embedded, clustered with constrained KMeans into balanced groups of 10 to 20 items, and each cluster is summarized by a reasoning LLM into a high-level edge case with a handling rule. Third, a final reasoning-LLM pass merges similar cluster-level cases so that the same phenomenon is not handled by multiple rules. The human expert then reviews clusters and rules in a dashboard, edits or rejects them, and triggers another iteration of annotation with the augmented codebook.

What would settle it

Run Co-DETECT on a training split of GabHateCorpus and GoEmotions, append the induced rules, and measure F1 on a held-out test split; if the F1 gain disappears or reverses on the held-out split, the reported improvement is a fit to the low-confidence cases used to create the rules rather than a generalizable codebook gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that descriptive, high-level, generalizable edge cases can be discovered collaboratively: the LLM supplies candidate cases through low confidence scores and explanations, a clustering-and-reasoning stage turns them into compact handling rules, and the human expert decides which rules enter the codebook. In the paper's framing, edge-case discovery is a mixed-initiative process rather than purely manual review or a fully automatic annotation pass. The companion quantitative experiment holds the annotator fixed and varies only the codebook, reporting F1 gains on all three evaluated tasks after one Co-DETECT iteration. A user study with 10 experts, plus a check in which 33 of 41 sampled cases (80.5%) were judged well-described by the suggested edge-case descriptions, is offered as evidence that the proposed cases are relevant and at the right level of abstraction.

Load-bearing premise

The central claim rests on the assumption that edge-case rules induced from a corpus's low-confidence examples improve classification on that same corpus in a way that also holds for new, unseen data, since the reported F1 gains use the same corpus for both rule induction and evaluation.

Editorial extensions

If this is right

  • A single iteration of Co-DETECT can augment a sketch-level codebook with compact handling rules that raise F1 on the same corpus, as shown in Table 1.
  • The workflow can surface edge cases beyond an expert's prior knowledge: 90% of user-study participants said Co-DETECT may help discover new edge cases.
  • The suggested rules are judged quickly: in the quantitative evaluation, an expert could decide whether a sample was covered by an edge-case description in under 5 seconds.
  • Because the annotator is held fixed and only the codebook changes, the F1 gains are attributed to codebook improvement rather than to a stronger model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's evaluation, the same pipeline could target other failure modes, such as label imbalance or domain shift, by changing what counts as 'low confidence' from a model confidence score to a calibration or uncertainty measure.
  • The generalization of the rules is under-tested; a natural extension would be to induce rules on one corpus and measure F1 on a held-out split, which would separate genuine rule discovery from in-sample fitting.
  • If edge-case rules prove stable across corpora, accumulated codebooks could become reusable domain assets that bootstrap future annotation tasks.
  • The 40% of participants who worried about missed edge cases points to a useful next metric: recall of edge cases, not just precision of the proposed ones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Co-DETECT, a mixed-initiative annotation framework in which an LLM annotates a user-provided corpus, flags low-confidence examples as potential edge cases, clusters and induces high-level rules with a reasoning LLM, and lets a human expert edit and append rules to a codebook. The paper reports a 10-participant user study, a small quantitative human check of edge-case descriptions, and F1 comparisons on GabHateCorpus and GoEmotions before and after one Co-DETECT iteration, concluding that the framework is effective.

Significance. If validated, Co-DETECT addresses a real bottleneck in codebook-based text annotation: the costly discovery and codification of edge cases. The backend pipeline (item-level flagging, constrained KMeans, and DeepSeek-R1 based induction and merging) is clearly described, and the public demo and code are concrete assets. However, the current validation is insufficient to support the paper's central claim that Co-DETECT discovers high-level, generalizable edge-case rules that improve codebook-driven classification, because the main quantitative experiment is evaluated on the same corpus used to induce the rules and the user study is small and partly author-included.

major comments (3)
  1. [Section 5 (Table 1)] The F1 comparison evaluates the augmented codebook on the same corpus from which the edge-case rules were induced (Step 1 flags low-confidence examples in the full corpus and Step 3 merges rules derived from those same examples), so the reported gains, including the GoEmotions-Positive jump from 0.0300 to 0.3297, may reflect in-sample fitting rather than generalizable codebook improvement; no held-out split, cross-validation, or out-of-distribution evaluation is reported. This is the central evidence for the claim that Co-DETECT produces high-level, generalizable rules, and the paper's own Broader Impact Statement acknowledges the analogous Clever Hans risk. A held-out split (or nested cross-validation) with confidence intervals and, ideally, a comparison against a human-only codebook revision baseline, is needed before the claim is supported.
  2. [Section 4] The user study has only 10 participants, 5 of whom are co-authors, and the quantitative human evaluation in Section 4.3 involves 3 participants and 41 self-selected samples; while these results are useful as pilot usability feedback, they do not provide evidence about downstream annotation quality (for example, whether the improved codebook changes annotator agreement or accuracy on unseen data). The paper should either soften the effectiveness claims or add a larger, preregistered evaluation with non-author participants and a measured annotation-quality outcome.
  3. [Section 3 (Steps 1-3)] The algorithm's behavior depends on several free parameters—the low-confidence threshold for flagging edge cases, the cluster-size bounds (10-20) for constrained KMeans, and the verbalized-confidence prompt—but no sensitivity analysis or ablation is reported, so it is unclear how stable the discovered rules are across reasonable settings. Reporting these ablations would strengthen the reproducibility of the central pipeline.
minor comments (5)
  1. [Abstract and Section 7] The phrases 'prove the effectiveness' overstate what the evidence supports; 'provide evidence for' or 'suggest' would be more accurate given the preliminary evaluation.
  2. [Section 5 (Table 1)] Table 1 does not report the number of annotated examples, standard deviations, or significance tests, and it is unclear whether the two iterations use identical test instances.
  3. [Section 5 and Appendix] The initial codebooks and the exact prompts used for GPT-4.1 and DeepSeek-R1 are not provided in the appendix, which hinders replication of Section 5.
  4. [Appendix A] One survey question refers to 'AutoDETECT' instead of Co-DETECT, which is inconsistent with the system name used elsewhere.
  5. [Figure 2] The screenshot in Figure 2 is difficult to read in the submitted PDF; enlarging the relevant panels or adding an annotated schematic would improve clarity.

Circularity Check

2 steps flagged · score 6.0 of 10

Table 1's F1 gains are in-sample: edge-case rules are induced from the same corpus on which the improved codebook is then evaluated.

  1. fitted input called prediction [Section 5, Table 1 (Can Improved Codebook Benefit Automatic Annotation?)]
    "we provide GPT-4.1 with codebooks before and after Co-DETECT enhancement and compare its classification F1, varying only the codebook. ... The results are exhibited in table 1, where we observe an increase in F1 scores across different datasets."

    The augmented codebook is built from edge-case rules induced from low-confidence examples in the same corpus: Step 1 flags low-confidence items, Step 2 clusters their case descriptions, and Step 3 merges cluster-level rules. Table 1 then measures F1 with the augmented codebook on that same corpus, with no held-out split or cross-validation reported. A rule such as 'ambiguous references to protected groups lack explicit demeaning language -> no hate speech' is derived from the very examples whose low confidence flagged them; re-applying it to those examples reassigns them by construction, so the F1 gain in Table 1 is an in-sample fit rather than a prediction on unseen data.

  2. other [Section 4.3, Quantitative Human Evaluation on Edge Case Validity]
    "Each participant was asked to randomly select 1 to 2 samples from each edge case cluster and manually assess how many were accurately captured by the Co-DETECT-suggested edge case descriptions. Among 41 randomly selected samples, 33 (80.5%) were reported as well-described by the suggested edge case descriptions."

    The suggested descriptions are generated from those same clusters in Step 2 ('Each cluster of [Case Description] ... are fed to DeepSeek-R1 to generate Cluster-wise Edge Cases'), so evaluating whether a description covers samples drawn from the very cluster it summarizes is a consistency check, not an independent test of generality. The paper uses this result to infer that the descriptions are 'sufficiently high-level to cover more than one samples,' but no unseen or held-out cluster is evaluated, so the high-level generalization claim is not independently supported.

full rationale

The paper is a system paper, not a formal derivation, so most of its claims are empirical rather than equation-level. The central quantitative claim, however, reduces to an in-sample comparison. Co-DETECT's Steps 1-3 induce edge-case handling rules from low-confidence examples in the target corpus; Section 5 then compares GPT-4.1's F1 with the original versus augmented codebook on that same corpus, with no held-out split or cross-validation. Because the added rules were fitted to the very examples whose confidence was low, the observed F1 increase is statistically forced and does not demonstrate that the codebook generalizes to unseen posts. The paper's own broader impact statement names exactly this Clever Hans risk. The user study (10 participants, half co-authors) is self-reported and small but not circular; the Section 4.3 validity check is also in-sample because participants pick samples from the same clusters the descriptions were induced from. No other load-bearing self-citation or imported uniqueness theorem appears. Overall, the headline improvement is partially circular/in-sample, warranting a 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The system has no mathematical derivation; its central claim rests on empirical assumptions about LLM behavior and human oversight. The key free parameter is the cluster size bounds, chosen empirically; the confidence threshold for edge case flagging is not reported. The main axioms are that LLM verbalized confidence identifies genuine edge cases, that reasoning LLMs can write general rules from clusters, that 500-1000 texts suffice, and that human review filters model biases. No new entities are introduced.

free parameters (3)
  • Cluster size bounds for constrained KMeans = 10-20
    Authors state clusters larger than 20 increase reasoning burden and hallucination, and smaller than 5 generate overspecific edge cases; these bounds are chosen empirically (Section 3, Step 2).
  • Low-confidence threshold for flagging edge cases = not reported
    The LLM reports edge cases when confidence is low, but the paper does not specify the threshold used; this is a free parameter of the method.
  • Verbalized confidence prompting = LLM-reported confidence scores
    Confidence scores from GPT-4.1 are used to identify edge cases; the calibration is assumed from prior work (Tian et al. 2023) and not re-validated here.
assumptions (4)
  • domain assumption LLM verbalized confidence scores are reliable indicators of annotation quality
    The entire edge-case discovery pipeline relies on low-confidence annotations flagging genuine ambiguous cases; invoked in Section 3 Step 1 with citation to Tian et al. (2023), not re-validated in this paper.
  • domain assumption Reasoning LLMs can induce high-level, generalizable edge case descriptions from clustered item-level cases
    DeepSeek-R1 is relied upon to summarize and merge edge cases (Section 3 Steps 2-3); the paper provides only a small human evaluation (41 samples) of the resulting descriptions.
  • domain assumption A corpus of 500-1000 texts is sufficient for representative edge case discovery
    The system suggests this input size to ensure representativeness (Section 2.1); no evidence is given that this range generalizes across domains.
  • domain assumption Human expert review prevents model-specific biases from entering the codebook
    The broader impact statement warns that rules may reinforce model-specific biases and says users should verify rules; the effectiveness of this human check is not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification." pith.science (2026). https://pith.science/paper/7LX4QRRV

@misc{pith2026250705010,
  author       = {Pith},
  title        = {Pith review of: Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7LX4QRRV}},
  note         = {Machine review of arXiv:2507.05010}
}
read the original abstract

We introduce Co-DETECT (Collaborative Discovery of Edge cases in TExt ClassificaTion), a novel mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models (LLMs). Co-DETECT starts with an initial, sketch-level codebook and dataset provided by a domain expert, then leverages the LLM to annotate the data and identify edge cases that are not well described by the initial codebook. Specifically, Co-DETECT flags challenging examples, induces high-level, generalizable descriptions of edge cases, and assists user in incorporating edge case handling rules to improve the codebook. This iterative process enables more effective handling of nuanced phenomena through compact, generalizable annotation rules. Extensive user study, qualitative and quantitative analyses prove the effectiveness of Co-DETECT.

Figures

Figures reproduced from arXiv: 2507.05010 by the authors.

Figure 1
Figure 1. traditional workflow of text annotation, where experts rely on their own or crowdworkers to identify edge cases and update codebook based on the discovered prevalent edge cases. Co-DETECT mixed-initiative workflow of edge case discovery, where LLMs propose prevalent and representative edge cases and the visual interface assists human expert to verify the proposed edge cases. velopment and the annotation process inte… view at source ↗
Figure 2
Figure 2. User Interface – Analysis Dashboard analysis stage section 2.2. 2.1 Preparation Stage Onboarding. To ensure a smooth onboarding expe￾rience, the first launch of Co-DETECT triggers an intro.js tour that guides users through the input and dashboard pages. Furthermore, users can also click “Load Demo Data” to explore a sample usecase in annotating hate speech from social media. Input Page. After familiarized with Co-DE… view at source ↗
Figure 3
Figure 3. Co-DETECT’s backend algorithm for automatically discover representative edge cases. Firstly, an LLM [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: User Interface – Homepage. • Have you used annotation support or guideline-generation tools previously? (Yes, No) (2) Expectation • What is your normal workflow of identifying edge cases in text annotation? • In the dataset that you plan to analyze with AutoDETECT, did…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 18 canonical work pages

  1. [1]

    David Autor. 2014. https://www.nber.org/papers/w20485 Polanyi's paradox and the shape of employment growth . Technical report, National Bureau of Economic Research

    work page 2014

  2. [2]

    Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. https://doi.org/10.1162/tacl_a_00449 Dealing with disagreements: Looking beyond the majority vote in subjective annotations . Transactions of the Association for Computational Linguistics, 10:92--110

    doi:10.1162/tacl_a_00449 2022

  3. [3]

    DeepSeek-AI. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

    arXiv 2025

  4. [4]

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. https://doi.org/10.18653/v1/2020.acl-main.372 G o E motions: A dataset of fine-grained emotions . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040--4054, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.372 2020

  5. [5]

    Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.acl-long.626 Is GPT -3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173--11195, Toronto, Canada. Association for Computat...

    doi:10.18653/v1/2023.acl-long.626 2023

  6. [6]

    Jakub Dr \'a pal, Hannes Westermann, Jaromir Savelka, et al. 2023. Using large language models to support thematic analysis in empirical legal studies. In JURIX, pages 197--206

    work page 2023

  7. [7]

    Zackary Okun Dunivin. 2024. https://arxiv.org/abs/2401.15170 Scalable qualitative coding with llms: Chain-of-thought reasoning matches human performance in some hermeneutic tasks . Preprint, arXiv:2401.15170

    arXiv 2024

  8. [8]

    James A Evans and Jacob G Foster. 2011. https://www.science.org/doi/abs/10.1126/science.1201765 Metaknowledge . Science, 331(6018):721--725

    doi:10.1126/science.1201765 2011

Show all 48 references
  1. [9]

    Yu Fan, Jingwei Ni, Jakob Merane, Etienne Salimbeni, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2025 a . htt...

    2025

  2. [10]

    Yu Fan, Yang Tian, Shauli Ravfogel, Mrinmaya Sachan, Elliott Ash, and Alexander Hoyle. 2025 b . https://arxiv.org/abs/2507.01234 The medium is not the message: Deconfounding text embeddings via linear concept erasure . Preprint, arXiv:2507.01234

    2025

  3. [11]

    Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.415 When the majority is wrong: Modeling annotator disagreement for subjective tasks . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6...

    2023 doi

  4. [12]

    Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. https://doi.org/10.18653/v1/2021.naacl-main.204 Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning . In Proceedings of the 2021 Conferenc...

    2021 doi

  5. [13]

    Lukas M Fuchs, Yu Fan, and Christian von Scheve. 2021. https://onlinelibrary.wiley.com/doi/full/10.1111/imig.12795 Value differences between refugees and german citizens: insights from a representative survey . International Migration, 59(5):59--81

    2021 doi

  6. [14]

    u gener, J \

    Andreas F \"u gener, J \"o rn Grahl, Alok Gupta, and Wolfgang Ketter. 2022. https://pubsonline.informs.org/doi/abs/10.1287/isre.2021.1079 Cognitive challenges in human--artificial intelligence collaboration: Investigating the path toward productive delegation . Information Sys...

    2022

  7. [15]

    Fabrizio Gilardi, Meysam Alizadeh, and Ma \"e l Kubli. 2023. https://www.pnas.org/doi/abs/10.1073/pnas.2305016120 Chatgpt outperforms crowd workers for text-annotation tasks . Proceedings of the National Academy of Sciences, 120(30):e2305016120

    2023 doi

  8. [16]

    Andrew Halterman and Katherine A. Keith. 2025. https://arxiv.org/abs/2407.10747 Codebook llms: Evaluating llms as measurement tools for political science concepts . Preprint, arXiv:2407.10747

    2025

  9. [17]

    Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024 a . https://doi.org/10.18653/v1/2024.naacl-industry.15 A nno LLM : Making large language models to be better crowdsourced annotators . In Proceeding...

    2024 doi

  10. [19]

    Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al. 2018. https://doi.org/10.31234/osf.io/hqjxn The gab hate corpus: A collection of 27k posts annotated ...

    2018 doi

  11. [20]

    Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Yehsong Kim, Kris Coombs, Gwenyth Portillo-Wightman, Shreya Havaldar, Elaine Gonzalez, Joseph Hoover, Aida Azatian, Gabriel Cardenas, Alyzeh Hussain, Austin Lara, Adam Omary, Christina Park, Xin Wang, Clarisa Wijaya, Y...

    2021

  12. [21]

    Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. 2024. https://aclanthology.org/2024.eacl-demo.18/ MEGA nno+: A human- LLM collaborative annotation system . In Proceedings of the 18th Conference of the European Chapter of the Association for Computatio...

    2024

  13. [22]

    Elisabeth Kirsten, Annalina Buckmann, Leona Lassak, Nele Borgert, Abraham Mhaidli, and Steffen Becker. 2025. https://arxiv.org/abs/2501.19275 From assistance to autonomy -- a researcher study on the potential of ai support for qualitative data analysis . Preprint, arXiv:2501.19275

    2025 arXiv

  14. [23]

    Lam, Janice Teoh, James A

    Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. 2024. https://doi.org/10.1145/3613904.3642830 Concept induction: Analyzing unstructured text with high-level concepts using lloom . In Proceedings of the CHI Conference on Human Factors in C...

    2024

  15. [24]

    Josh Levy-Kramer. 2018. https://github.com/joshlk/k-means-constrained k-means-constrained

    2018

  16. [25]

    Rajiv Movva, Kenny Peng, Nikhil Garg, Jon Kleinberg, and Emma Pierson. 2025. https://arxiv.org/abs/2502.04382 Sparse autoencoders for hypothesis generation . Preprint, arXiv:2502.04382

    2025 arXiv

  17. [26]

    Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, and Elliott Ash. 2025 a . https://arxiv.org/abs/2506.19467 Can large language models capture human annotator disagreements? Preprint, arXiv:2506.19467

    2025

  18. [27]

    Jingwei Ni, Tobias Schimanski, Meihong Lin, Mrinmaya Sachan, Elliott Ash, and Markus Leippold. 2025 b . https://aclanthology.org/2025.naacl-long.271/ DIRAS : Efficient LLM annotation of document relevance for retrieval augmented generation . In Proceedings of the 2025 Conferen...

    2025

  19. [28]

    Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, and Markus Leippold. 2024. https://doi.org/10.18653/v1/2024.acl-long.104 AF a CTA : Assisting the annotation of factual claim detection with reliable LLM annotators . In Proceedings of the 62nd Annual Me...

    2024 doi

  20. [29]

    OpenAI . 2024. New embedding models and api updates: text-embedding-3-large. https://openai.com/index/new-embedding-models-and-api-updates/. Introduced text‑embedding‑3‑large (3072‑dim), offers strongest performance on MIRACL (54.9

    2024

  21. [30]

    OpenAI . 2025 a . Introducing gpt‑4.1, gpt‑4.1 mini & nano. https://openai.com/index/gpt-4-1/. Released April 14, 2025; includes full, mini, and nano variants (1M token context window; optimized for coding and instruction‑following)

    2025

  22. [31]

    OpenAI . 2025 b . https://openai.com/index/introducing-o3-and-o4-mini Introducing openai o3 and o4‑mini . O3 is a reasoning‑focused generative model with advanced capabilities in coding, math, and visual perception; system card provides detailed benchmarks and safety evaluations

    2025

  23. [32]

    Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. https://api.semanticscholar.org/CorpusID:259000016 Automated annotation with generative ai requires validation . ArXiv, abs/2306.00176

    2023 arXiv

  24. [33]

    Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer. 2024. https://doi.org/10.18653/v1/2024.naacl-long.164 T opic GPT : A prompt-based topic modeling framework . In Proceedings of the 2024 Conference of the North American Chapter of the Association for ...

    2024 doi

  25. [34]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language ...

    2023 doi

  26. [35]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023 b . https://arxiv.org/abs/2305.14975 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-...

    2023 arXiv

  27. [36]

    Petter T \"o rnberg. 2024. https://api.semanticscholar.org/CorpusID:267547980 Best practices for text annotation with large language models . ArXiv, abs/2402.05129

    2024 arXiv

  28. [37]

    Petter Törnberg. 2023. https://arxiv.org/abs/2304.06588 Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning . Preprint, arXiv:2304.06588

    2023 arXiv

  29. [38]

    Vijay Viswanathan, Kiril Gashteovski, Kiril Gashteovski, Carolin Lawrence, Tongshuang Wu, and Graham Neubig. 2024. https://doi.org/10.1162/tacl_a_00648 Large language models enable few-shot clustering . Transactions of the Association for Computational Linguistics, 12:321--333

    2024 doi

  30. [39]

    Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. https://doi.org/10.1145/3613904.3641960 Human-llm collaborative annotation through effective verification of llm labels . In Proceedings of the 2024 CHI Conference on Human Factors in Computing Sys...

    2024

  31. [40]

    Zihan Wang, Jingbo Shang, and Ruiqi Zhong. 2023. https://arxiv.org/abs/2305.13749 Goal-driven explainable clustering via language descriptions . Preprint, arXiv:2305.13749

    2023 arXiv

  32. [41]

    John Wilkerson and Andreu Casas. 2017. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2968080 Large-scale computerized text analysis in political science: Opportunities and challenges . Annual Review of Political Science, 20(1):529--544

    2017

  33. [42]

    Ziang Xiao, Xingdi Yuan, Q Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer. 2023. https://dl.acm.org/doi/abs/10.1145/3581754.3584136 Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding . In Companion proceedings o...

    2023

  34. [43]

    Tian Xie and Xueru Zhang. 2024. https://arxiv.org/abs/2405.08027 Automating data annotation under strategic human agents: Risks and potential solutions . Preprint, arXiv:2405.08027

    2024 arXiv

  35. [44]

    Yuwei Zhang, Zihan Wang, and Jingbo Shang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.858 C luster LLM : Large language models as a guide for text clustering . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13903--13920,...

    2023 doi

  36. [45]

    Chengshuai Zhao, Zhen Tan, Chau-Wai Wong, Xinyan Zhao, Tianlong Chen, and Huan Liu. 2025. https://arxiv.org/abs/2502.10937 SCALE : Towards collaborative content analysis in social science with large language model agents and human intervention . Preprint, arXiv:2502.10937

    2025

  37. [46]

    Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/7e810b2c75d69be186cadd2fe3febeab-Paper-Conference.pdf Goal driven discovery of distributional differences via language descriptions...

    2023

  38. [47]

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. https://direct.mit.edu/coli/article/50/1/237/118498 Can large language models transform computational social science? Computational Linguistics, 50(1):237--291

    2024

  39. [48]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.