Are Large Language Models the future crowd workers of Linguistics?

Iris Ferrazzo

REVIEW 4 major objections 7 minor 13 references

Are Large Language Models the future crowd workers of Linguistics?

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that GPT-4o-mini, prompted with zero-shot instructions, can replace human informants in forced-choice linguistic experiments, matching or exceeding human accuracy in two replicated studies.

desk verdict A useful, transparent pilot undermined by an unsupported independence assumption and absent inferential statistics; worth referee effort, but the 'outperforms humans' claim needs reframing. read the letter →

arxiv 2502.10266 v1 pith:FT4DC4AZ submitted 2025-02-14 cs.CL cs.AI

Iris Ferrazzo This is my paper · ORCID

classification cs.CLcs.AI

keywords largelanguagemodelscrowdsourcingempiricallinguisticszero-shotpromptingchain-of-thoughtGPT-4o-minigenderassignmentneologismdetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a large language model can serve as the crowd worker of empirical linguistics, taking over data elicitation normally done by human participants. It replicates two forced-choice tasks—Spanish–English gender assignment in code-switched sentences and French neologism detection—using GPT-4o-mini with zero-shot prompts. The paper's headline finding is that the model outperforms human informants in all experimental conditions tested, completing the tasks in about a minute per informant instead of the 20–50 minutes reported for people. It presents this as evidence that LLM-based synthetic informants are a viable, low-cost, accessible complement or substitute to human crowdsourcing, with prompt design (notably Chain-of-Thought) as a way to steer the model toward human-like performance.

What carries the argument

The machinery is a replication wrapper around the model: it reads the original stimuli and filler sentences, sends each item as a zero-shot prompt to GPT-4o-mini, treats each separate API run as one 'LLM-informant,' repeats the run as many times as there were human participants in the original study, and scores the synthetic answers against the original human baseline. The system prompt impersonates a participant profile—for instance, a Spanish–English bilingual or a native French speaker—and demands a forced binary choice. Two follow-up modifications isolate the effect of prompt design: a strengthened system-role instruction and Chain-of-Thought prompting with worked examples.

What would settle it

Repeat the exact zero-shot prompt on the same items many times and compare the response distribution with the between-participant distribution in the original human data; if the model is near-deterministic or its run-to-run variance is far smaller than human variance, the 'LLM-informant' assumption fails and the claimed outperformance is not evidence about human–model equivalence.

Watch

Extended reading notes

Core claim

The paper's central claim is that GPT-4o-mini, queried once per 'LLM-informant' with the same zero-shot prompt, produces responses that align with—and on the main comparison exceed—human performance on two published linguistics experiments. Its stated headline finding is that the model outperforms human informants in all experimental conditions tested. In the gender-assignment study, the model reproduces the human pattern of gender congruency—semantic gender for human-denoting nouns, a masculine default for inanimate nouns, and facilitation from morphological cues—with higher congruency scores than humans. In the neologism-detection study, the model reaches a mean accuracy of 0.985 against 0.91 for humans, although it also shows a yes-bias on filler items; a Chain-of-Thought follow-up raises filler accuracy from 0.77 to 0.99, and the paper notes that in the CoT condition humans slightly outperform the model on one condition (0.92 vs 0.91) while the two scores are close. The paper concludes that the model is a viable basis for synthetic crowdsourcing, while cautioning that over-performance can make it a poor substitute when the research question targets human-specific variation.

Load-bearing premise

The result stands on treating each separate GPT-4o-mini API call as an independent human-like informant; if repeated calls are not independent draws from a participant population, then the reported alignment scores do not measure human–model comparability.

Editorial extensions

If this is right

A zero-shot LLM pipeline can reproduce at least two kinds of forced-choice linguistics experiments, a discourse-completion task and a metalinguistic judgment task, without human participants.
In gender assignment, the model replicates the main human tendencies (semantic gender, masculine default, morphological cues), suggesting synthetic responses could pre-test stimuli or generate hypotheses before human data collection.
Chain-of-Thought prompting corrects the model's yes-bias on filler items, raising filler accuracy from 0.77 to 0.99, so prompt design can tune the model between raw accuracy and human-like performance.
Reaction times recorded from LLM runs are not a meaningful cognitive measure, so replications involving response-time comparisons with humans need a different evaluation method.
The framework is lightweight and reusable by non-programmers, extending the potential use of LLMs beyond NLP annotation to humanities research workflows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that its 'LLM-informants' are not independent in a statistical sense: every run shares the same model weights, prompt template, and training data, so run-to-run variance likely underestimates human population variance and the reported outperformance may reflect a single frozen response tendency rather than population-level competence.
Because the model's high accuracy can erase the between-participant differences that linguistic experiments are designed to measure, a more cautious use would be as a pilot or sensitivity-analysis tool—testing whether stimuli are ambiguous—rather than as a full replacement for human participants.
The filler error pattern suggests a general yes-bias in instruction-tuned models; this is testable across other models and tasks, and CoT prompting may be a general corrective whenever negative responses are required.
The replication design could be extended to open-source models and to more Romance languages to separate genuine linguistic generalization from memorized training-data patterns, a comparison the paper names as future work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

You should know two things about this paper. First, it is a genuinely useful pilot: it takes two real empirical linguistics studies, reproduces their forced-choice tasks with zero-shot GPT-4o-mini, and discloses all prompts and code. That is exactly the kind of transparent exploratory work that helps map where LLMs might fit in data collection. Second, the headline claim that the model 'outperforms human informants in all experimental conditions' is not backed by any statistical inference, and the deeper issue is that the paper treats each API call as an independent 'informant' without supporting that assumption.

What is new is applying LLM-as-participant to linguistic tasks that go beyond annotation: gender assignment in code-switching and neologism detection. The paper does a good job of describing the pipeline, showing the prompts, and honestly discussing the model's over-performance and its failure on filler items. The Chain-of-Thought follow-up is a sensible attempt to address the filler problem. The limitations section is candid about the small scale.

The problems are real but concentrated. First, no confidence intervals, error bars, or significance tests anywhere. The reader is asked to accept that means like 0.88 vs 0.55 indicate superiority. Second, and more load-bearing, the independence assumption in Section 3.3. The paper says that because each run is a separate API call, the answers are unrelated and contamination is prevented. But all runs share the same model weights, prompt template, and decoding defaults. They are correlated draws from a narrow stochastic process, not independent samples from a human-like population. A homogeneous model that always picks the majority option will mechanically beat a human mean diluted by inter-individual variation, while carrying none of the variance the original experiments aim to measure. The paper's own discussion of 'over-performance' concedes this but does not connect it back to the independence assumption. The post-hoc exclusion of 'velours' is also a minor red flag.

To be fair, the stress-test's claim that the runs collapse to 'one agent' is overstated—temperature should produce some spread—but the core objection stands. The abstract and finding list say 'outperforms', while Section 6 says LLMs 'do not consistently serve as suitable substitutes.' That internal tension needs resolving.

Bottom line: the paper deserves a serious referee, because the question is timely, the methods transparent, and the flaws fixable. A revision should add inferential statistics, interrogate the independence assumption, and reframe the conclusion as 'comparable in some conditions, with important caveats' rather than 'outperforms.' I would not cite the current results as evidence of human-LLM equivalence, but I would cite it as a transparent pilot and a cautionary example of how not to count API calls as participants.

Recommendation: send it to review, with the expectation of heavy revision.

Referee Report

4 major / 7 minor

Summary. The paper proposes using OpenAI's GPT-4o-mini as a synthetic crowd worker in empirical linguistics. It replicates two forced-choice studies—Cruz (2023) on gender assignment in Spanish–English code-switching and Lombard et al. (2021) on neologism detection in French—by zero-shot prompting the model with participant profiles and task instructions. Each API call is treated as one 'LLM-informant,' and the model's aggregate response rates (gender congruency and detection accuracy) are compared with the published human means. The paper reports that GPT-4o-mini outperforms human informants in all experimental conditions, then adds two follow-up experiments using role prompting and chain-of-thought prompting to address the model's poor performance on filler items. The authors argue that the proposed pipeline is adaptable, accessible to non-programmers, and points toward a possible interdisciplinary opening between NLP and the humanities.

Significance. If substantiated, the claim that a zero-shot LLM pipeline can reproduce or exceed human performance in linguistically motivated forced-choice tasks would be a valuable proof of concept for synthetic crowdsourcing, particularly for Romance languages that are under-represented in this literature. The paper has concrete strengths: the code is released, the prompts are documented in full in Appendix C, the two case studies go beyond simple labeling tasks, and the authors explicitly discuss the risk of model over-performance. However, the headline result currently rests on descriptive mean comparisons without uncertainty quantification, and the central assumption that separate API calls simulate independent human informants is unexamined. The significance of the result is therefore not yet established; the paper is better read as a promising pilot than as evidence that LLMs can replace human participants.

major comments (4)

[Section 6; Figures 3 and 4] The central claim that 'GPT-4o-mini outperforms human informants in all experimental conditions tested' is not statistically supported. Figures 3 and 4 plot single mean values for humans and the model with no error bars, confidence intervals, significance tests, or effect sizes, and the text in Sections 4.2 and 5.2 reports only aggregate M-human and M-GPT values. The statement in Section 3.3 that the pipeline 'is repeated twice per replicated study to ensure cross-validation and statistical relevance' does not make n=2 runs a basis for inference. The authors should report human variability from the original studies wherever possible, compute bootstrap or other confidence intervals for the LLM means, and use tests that account for item and participant clustering before making the outperformance claim.
[Section 3.3] The load-bearing assumption that separate GPT-4o-mini API calls constitute independent and diverse 'LLM-informants' is unsupported. All runs share the same model weights, prompt template, decoding defaults, and training data, so the repeated calls are correlated realizations of one stochastic process rather than independent draws from a human-like population. If the model is near-deterministic at the default temperature, the variance across synthetic informants will vastly underestimate human population heterogeneity, making the comparison one between a single homogeneous model response and a heterogeneous human average. The paper should either provide evidence of between-call response diversity (temperature settings, response entropy, item-level variance across runs) or reframe the claims as model-level behavior rather than simulated crowds.
[Section 5.2.2; Appendix C] The chain-of-thought follow-up is presented as demonstrating 'higher alignment to human performance,' but the CoT prompt was constructed after observing the baseline's poor filler accuracy, and its two examples encode the desired response pattern ('non' for a known-word sentence). Because the same test data were used to select and tune this prompt, the reported filler improvement is a post hoc fit rather than an independent evaluation of CoT prompting as a general strategy. This finding should be labeled exploratory, and the authors should either validate CoT on a held-out dataset or report the prompt-selection process explicitly as a limitation.
[Section 5 and Section 5.2] The second replication discards reaction time as 'not a reliable parameter' for the model, but reaction time was one of the two dependent measures in Lombard et al. (2021). Reporting only detection accuracy means the LLM pipeline does not actually reproduce the full experimental outcome of the original study, only a subset of it. The paper should state this explicitly as a limitation and narrow the claim of 'replication' accordingly.

minor comments (7)

[Section 3.1 heading] The heading 'GPT-4-mini as crowd worker' is inconsistent with the model name used throughout the rest of the paper, GPT-4o-mini; please correct it.
[Section 7] The word 'overperfom' is a typo for 'overperform'; please proofread the manuscript for such errors.
[Appendix C] The French prompt cells contain English translations inside the same table cell (e.g., 'partecipating'), which could be mistaken for part of the actual prompt; the translations should be separated into a distinct column or footnote.
[Figures 3 and 4] The numeric labels in Figure 4 are not clearly keyed to the four conditions and two groups in the caption; please add a legend or restructure the figure so readers can map each value to a condition.
[References] Several references are incomplete or formatted as author-year citations without full bibliographic entries (e.g., Ahmadabadi et al., Carvalho, Sheehan et al., Kocoń et al.); please complete these entries.
[Section 3.3] The phrase 'to ensure cross-validation and statistical relevance' is misleading for two repeated runs of the same pipeline; 'cross-validation' is a model-selection procedure, and two runs do not provide statistical relevance. Please rephrase this sentence.
[Section 4.2] The comparison of task duration (72 seconds for the model vs. 25 minutes for humans) should clarify that the model time is API wall-clock time, not a psycholinguistic measure, so that readers do not interpret this as evidence about human-like processing speed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline result is an external comparison against published human data, and the follow-up prompt adjustments are post hoc but not definitionally tied to the reported outcome.

full rationale

The paper's central claim—that GPT-4o-mini outperforms human informants in the two replicated forced-choice tasks—is evaluated against external human data from Cruz (2023) and Lombard et al. (2021). The zero-shot baseline responses are generated by the model independently of those human means and then compared; no parameter is fitted to the human benchmark, and no equation defines the model's score in terms of the human score. The number of LLM runs is matched to the original participant counts, but that is a design choice, not a definitional link. The Section 3.3 assertion that separate API calls yield unrelated 'LLM-informants' is methodologically fragile, but it is a statistical-validity concern rather than a circularity: the independence assumption is not used to derive the outcome from the input. The CoT follow-up is post hoc: the prompt was modified after observing baseline filler errors, and the prompt contains a worked filler example demonstrating a 'no' response. However, the paper frames this as an adjustment ('we attempt to adjust the model's performance through two follow-up experiments'), not as an independent prediction, and the example items are excluded from the test data. Even if this limits the evidential value of the CoT result, it does not make the headline claim circular. No load-bearing step reduces to a fitted parameter renamed as a prediction, a self-citation chain, an imported uniqueness theorem, or a definitional equivalence. Therefore the derivation is self-contained with respect to circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce any new theoretical entities or fitted numerical parameters. Its central evidence depends on assumptions about task representativeness, the validity of published human means as benchmarks, the independence of repeated API calls, and the neutrality of the CoT examples. The first two are common in replication work, but the third is particularly fragile because LLM responses from the same model are not independent samples from a human population.

assumptions (4)

domain assumption The two replicated studies (Cruz 2023 and Lombard et al. 2021) are representative of empirical linguistic research designs.
The paper generalizes from two tasks to a broad claim about LLMs as future crowd workers in linguistics. This assumption is stated in Section 3.2, where the studies are selected based on Romance language focus, generation/understanding tasks, and linguistic grounding.
domain assumption Published aggregate human performance from the original studies is a valid benchmark for comparing LLM responses.
The evaluation in Section 3.3 uses the original papers' reported means as the human baseline. The author does not have access to the raw human data, so no variance-based comparison is possible.
ad hoc to paper Separate GPT-4o-mini API calls produce responses independent and diverse enough to simulate distinct human informants.
Section 3.3 claims that because runs are not linked, 'the answers are unrelated between LLM-informants and contamination is prevented.' This treats API sampling as equivalent to sampling human participants, which is a strong assumption about model stochasticity and diversity.
ad hoc to paper The chain-of-thought prompting examples do not bias the model because they are excluded from the test dataset.
In Section 5.2.2, two example sentences are shown in the prompt and the author states they are excluded from the test set. This assumes no leakage or anchoring effect from the examples on the remaining items.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Are Large Language Models the future crowd workers of Linguistics?." pith.science (2026). https://pith.science/paper/FT4DC4AZ

@misc{pith2026250210266,
  author       = {Pith},
  title        = {Pith review of: Are Large Language Models the future crowd workers of Linguistics?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FT4DC4AZ}},
  note         = {Machine review of arXiv:2502.10266}
}

read the original abstract

Data elicitation from human participants is one of the core data collection strategies used in empirical linguistic research. The amount of participants in such studies may vary considerably, ranging from a handful to crowdsourcing dimensions. Even if they provide resourceful extensive data, both of these settings come alongside many disadvantages, such as low control of participants' attention during task completion, precarious working conditions in crowdsourcing environments, and time-consuming experimental designs. For these reasons, this research aims to answer the question of whether Large Language Models (LLMs) may overcome those obstacles if included in empirical linguistic pipelines. Two reproduction case studies are conducted to gain clarity into this matter: Cruz (2023) and Lombard et al. (2021). The two forced elicitation tasks, originally designed for human participants, are reproduced in the proposed framework with the help of OpenAI's GPT-4o-mini model. Its performance with our zero-shot prompting baseline shows the effectiveness and high versatility of LLMs, that tend to outperform human informants in linguistic tasks. The findings of the second replication further highlight the need to explore additional prompting techniques, such as Chain-of-Thought (CoT) prompting, which, in a second follow-up experiment, demonstrates higher alignment to human performance on both critical and filler items. Given the limited scale of this study, it is worthwhile to further explore the performance of LLMs in empirical Linguistics and in other future applications in the humanities.

Figures

Figures reproduced from arXiv: 2502.10266 by the authors.

**Figure 2.** Replication pipeline. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Comparison of mean responses (M) in terms of gender [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Comparison of mean responses (M) in terms of neologism [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages

[1]

Design and evaluation of crowdsourcing platforms based on users’ confidence judgments

Samin Nili Ahmadabadi, Maryam Haghifam, Vahid Shah-Mansouri, and Sara Ershadmanesh. Design and evaluation of crowdsourcing platforms based on users’ confidence judgments. 14(1):18379. ISSN 2045-2322. doi:10.1038/s41598- 024-65892-7. URL https://www.nature.com/articles/s41598-024-65892-7 . Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohamma...

doi:10.1038/s41598-
[2]

The challenge of using LLMs to simulate human behavior: A causal inference perspective

George Gui and Olivier Toubia. The challenge of using LLMs to simulate human behavior: A causal inference perspective. doi:10.48550/ARXIV .2312.15524. URLhttps://arxiv.org/abs/2312.15524. Publisher: arXiv Version Number:

doi:10.48550/arxiv.2312.15524
[3]

On the role of large language models in crowdsourcing misinformation assessment

Jiechen Xu, Lei Han, Shazia Sadiq, and Gianluca Demartini. On the role of large language models in crowdsourcing misinformation assessment. 18:1674–1686. ISSN 2334-0770, 2162-3449. doi:10.1609/icwsm.v18i1.31417. URL https://ojs.aaai.org/index.php/ICWSM/article/view/31417. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can...

work page doi:10.1609/icwsm.v18i1.31417
[4]

Chatgpt in and for second language acquisition: A call for systematic research

ZhaoHong Han. Chatgpt in and for second language acquisition: A call for systematic research. 46(2):301–306. ISSN 0272-2631, 1470-1545. doi:10.1017/S0272263124000111. URL https://www.cambridge.org/core/product/ identifier/S0272263124000111/type/journal_article. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu...

arXiv
[5]

The promises and pitfalls of LLM annotations in dataset labeling: a case study on media bias detection

Tomas Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, and Timo Spinde. The promises and pitfalls of LLM annotations in dataset labeling: a case study on media bias detection. URL https://arxiv.org/abs/2411.11081. Version Number:

arXiv
[6]

ChatGPT vs

Lidiia Ostyakova, Veronika Smilga, Kseniia Petukhova, Maria Molchanova, and Daniel Kornev. ChatGPT vs. crowd- sourcing vs. experts: Annotating open-domain conversations with speech functions. In Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue , pages 242–254. Association for Computational Linguistics. doi:10.18653/v...

arXiv
[7]

Crowdsourc- ing lexical diversity

11 Are Large Language Models the future crowd workers of Linguistics? A PREPRINT Hadi Khalilia, Jahna Otterbacher, Gabor Bella, Rusma Noortyani, Shandy Darma, and Fausto Giunchiglia. Crowdsourc- ing lexical diversity. URLhttps://arxiv.org/abs/2410.23133. Version Number:

work page arXiv
[8]

ChatGPT: Jack of all trades, master of none

Jan Koco´n, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Koco ´n, Bartłomiej Koptyra, Wiktoria Mieleszczenko- Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łukasz Radli ´nski, Konrad Wojtasik, Stanisław Wo´ zniak, and Przemysław Kazie...

arXiv

Show all 13 references

[10]

Direct and indirect annotation with generative AI: A case study into finding animals and plants in historical text

Arjan Van Dalfsen, Folgert Karsdorp, Ayoub Bagheri, Dieuwertje Mentink, Thirza van Engelen, and Els Stronks. Direct and indirect annotation with generative AI: A case study into finding animals and plants in historical text. volume CEUR Workshop Proceedings. URL https://ceur-w...
[11]

Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, Jenny T. Liang, Ryan Liu, Ihita Mandal, Jeremiah Milbauer, Xiaolin Ni, Namrata Padmanabhan, Subhashini Ramkumar, Alexis Sudjianto, Jordan T...

doi arXiv
[2023]

doi:10.1017/pan.2023.2

ISSN 1476-4989. doi:10.1017/pan.2023.2. URL http://dx.doi.org/10.1017/pan.2023.2. Janos Borst, Jannis Klähn, and Manuel Burghardt. Death of the dictionary? – the rise of zero-shot sentiment classification. volume CEUR Workshop Proceedings. URL https://ceur-ws.org/Vol-3558/pape...

2023
[2024]

Andres Karjus and Christine Cuskley

URL https://arxiv.org/abs/2309.14379. Andres Karjus and Christine Cuskley. Evolving linguistic divergence on polarizing social media. Humanit. Soc. Sci. Commun., 11(1), March

arXiv
[9312]

URL https://direct.mit.edu/coli/article/50/1/237/118498/ Can-Large-Language-Models-Transform-Computational

doi:10.1162/coli_a_00502. URL https://direct.mit.edu/coli/article/50/1/237/118498/ Can-Large-Language-Models-Transform-Computational . 13 Are Large Language Models the future crowd workers of Linguistics? A PREPRINT A Appendix A: Supplementary materials The Python notebook tha...

doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Design and evaluation of crowdsourcing platforms based on users’ confidence judgments

Samin Nili Ahmadabadi, Maryam Haghifam, Vahid Shah-Mansouri, and Sara Ershadmanesh. Design and evaluation of crowdsourcing platforms based on users’ confidence judgments. 14(1):18379. ISSN 2045-2322. doi:10.1038/s41598- 024-65892-7. URL https://www.nature.com/articles/s41598-024-65892-7 . Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohamma...

doi:10.1038/s41598-

[2] [2]

The challenge of using LLMs to simulate human behavior: A causal inference perspective

George Gui and Olivier Toubia. The challenge of using LLMs to simulate human behavior: A causal inference perspective. doi:10.48550/ARXIV .2312.15524. URLhttps://arxiv.org/abs/2312.15524. Publisher: arXiv Version Number:

doi:10.48550/arxiv.2312.15524

[3] [3]

On the role of large language models in crowdsourcing misinformation assessment

Jiechen Xu, Lei Han, Shazia Sadiq, and Gianluca Demartini. On the role of large language models in crowdsourcing misinformation assessment. 18:1674–1686. ISSN 2334-0770, 2162-3449. doi:10.1609/icwsm.v18i1.31417. URL https://ojs.aaai.org/index.php/ICWSM/article/view/31417. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can...

work page doi:10.1609/icwsm.v18i1.31417

[4] [4]

Chatgpt in and for second language acquisition: A call for systematic research

ZhaoHong Han. Chatgpt in and for second language acquisition: A call for systematic research. 46(2):301–306. ISSN 0272-2631, 1470-1545. doi:10.1017/S0272263124000111. URL https://www.cambridge.org/core/product/ identifier/S0272263124000111/type/journal_article. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu...

arXiv

[5] [5]

The promises and pitfalls of LLM annotations in dataset labeling: a case study on media bias detection

Tomas Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, and Timo Spinde. The promises and pitfalls of LLM annotations in dataset labeling: a case study on media bias detection. URL https://arxiv.org/abs/2411.11081. Version Number:

arXiv

[6] [6]

ChatGPT vs

Lidiia Ostyakova, Veronika Smilga, Kseniia Petukhova, Maria Molchanova, and Daniel Kornev. ChatGPT vs. crowd- sourcing vs. experts: Annotating open-domain conversations with speech functions. In Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue , pages 242–254. Association for Computational Linguistics. doi:10.18653/v...

arXiv

[7] [7]

Crowdsourc- ing lexical diversity

11 Are Large Language Models the future crowd workers of Linguistics? A PREPRINT Hadi Khalilia, Jahna Otterbacher, Gabor Bella, Rusma Noortyani, Shandy Darma, and Fausto Giunchiglia. Crowdsourc- ing lexical diversity. URLhttps://arxiv.org/abs/2410.23133. Version Number:

work page arXiv

[8] [8]

ChatGPT: Jack of all trades, master of none

Jan Koco´n, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Koco ´n, Bartłomiej Koptyra, Wiktoria Mieleszczenko- Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łukasz Radli ´nski, Konrad Wojtasik, Stanisław Wo´ zniak, and Przemysław Kazie...

arXiv