Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Evaluating Large Language Models as Expert Annotators

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Material fingerprints replace non-convex optimization

desk verdict The submitted full text is a different paper from the abstract: a proof-of-concept for nearest-neighbor material model discovery, with one genuine normalization insight but a load-bearing uniqueness assumption that the paper's own 5%-noise results already contradict. read the letter →

arxiv 2508.07827 v1 pith:7SNY5KZC submitted 2025-08-11 cs.CL

Yu-Min Tseng , Wei-Lin Chen , Chung-Chi Chen , Hsin-Hsi Chen This is my paper
classification cs.CL
keywords materialfingerprintingmodeldiscoveryhyperelasticitylookuptablepatternrecognitionfull-fieldmeasurementsinverseproblemsstrainenergydensity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that identifying the constitutive model of a material—its stored-energy function and parameters—can be reduced to a nearest-neighbor lookup. Run a standardized experiment, record the response as a fingerprint, then compare it to a precomputed database of simulated fingerprints, each tagged with its generating model. In numerical benchmarks the method recovers the exact strain-energy function at zero noise and retains $R^2 > 0.98$ at 5% noise, without ever solving an optimization problem. The benefit is speed and robustness: the expensive part is done once, offline, and online identification is a pattern-match. The paper demonstrates this for hyperelastic materials under homogeneous and heterogeneous deformation, and argues the concept transfers to other material classes and experimental designs.

What carries the argument

The material fingerprint is a vector of responses measured under a standardized loading protocol—stress components in the supervised case, boundary forces and surface displacements in the unsupervised case. The database stores fingerprints alongside the strain energy density functions that generated them. Pattern recognition—in the reported implementation a nearest-neighbor search—selects the database entry with the closest normalized fingerprint. Two normalization steps carry the argument: fingerprint scaling makes responses from different specimens and parameter magnitudes comparable, and parameter rescaling means a single database fingerprint can represent all members of a one-parameter f

What would settle it

Take two models known to be nearly equivalent over the loading range—for instance Neo-Hooke and Ogden with exponent near 2—and generate their fingerprints under the same biaxial plate-with-hole protocol at zero noise. If the normalized fingerprints are closer than the noise level implied by the paper's 5% tests (e.g., separated by only a few percent), the uniqueness assumption fails and a lookup cannot guarantee the correct model. A complementary test would be to record fingerprints from a material whose true model is deliberately excluded from the database and check whether the method returns

Watch

Extended reading notes

Core claim

The central discovery is that material model identification can be framed as a database retrieval problem rather than an inverse optimization problem. Given a standardized experimental protocol, the paper defines a material fingerprint—normalized responses such as stress values, boundary reaction forces, or surface displacements—and builds a database by simulating every candidate model and parameter combination. For a new material, measuring the same protocol and applying pattern recognition yields the database entry whose fingerprint is closest; the tag of that entry is the discovered model. The paper shows that in both a 'supervised' setting (homogeneous deformation giving direct stress-st

Load-bearing premise

The method assumes that the standardized experimental protocol produces a response pattern unique to each distinct material model, so a measured fingerprint could not come from two different stored models; it also assumes the database contains a model form close to the true material, because the lookup can only return what is stored.

Editorial extensions

If this is right

  • Material characterization could become near-instant in the online phase for any material whose behavior lies within the database's model family, since no forward simulation or gradient optimization is needed.
  • The same fingerprint database can serve both direct stress-strain tests and indirect full-field tests, unifying two previously separate identification practices.
  • Because only physically admissible models are stored, the method cannot output a physically meaningless fit, a risk in free-form symbolic regression or neural-network constitutive models.
  • If the method generalizes to other material classes, the offline/online split opens the way for shared, continuously updated model libraries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The database-bound nature implies that the biggest risk is not noise but coverage: any material whose true response cannot be closely approximated by a stored model will be silently mapped to the nearest stored one, so the online match should be accompanied by a 'none of the above' threshold on fingerprint distance if the method is to serve as a genuine discovery tool.
  • The acknowledged equivalence between Neo-Hooke and Ogden models at small stretches, and the 5%-noise misidentification of Neo-Hooke as Demiray, suggest that fingerprints are not unique near model boundaries; a statistical matching rule that returns a set of models consistent within the noise level would better represent the actual information content.
  • The offline database generation cost scales with the number of models and parameter samples; database compression or active selection of experiments would be needed to make the approach practical for higher-dimensional parameter spaces.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The full text of the manuscript proposes Material Fingerprinting, a database-driven method for identifying hyperelastic material models without solving inverse optimization problems. In the offline phase, a database of material fingerprints is generated by simulating a standardized experiment for a library of candidate strain-energy functions over a parameter grid. In the online phase, a measured fingerprint is compared with the database using a nearest-neighbor pattern recognition algorithm, returning both the functional form and material parameters. The method is tested on two settings: supervised fingerprints from homogeneous uniaxial tension and simple shear stress data, and unsupervised fingerprints from a plate-with-hole biaxial experiment with displacement and reaction-force measurements. On synthetic data with 0%, 1%, and 5% noise, the paper reports exact recovery at 0% noise and R2 > 0.98 in all noisy cases. The central claim is that each material has a unique fingerprint, so model identification reduces to a lookup. A serious submission-level issue is that the title and abstract describe an unrelated paper on evaluating LLMs as expert annotators, while the body is entirely about material fingerprinting.

Significance. If the central claim were established, the approach would be an appealing alternative to classical inverse parameter identification: it shifts computational cost offline and avoids non-convex optimization in the online phase. The use of both homogeneous and heterogeneous deformation experiments, and the explicit treatment of noise levels, are useful contributions to the proof-of-concept. However, the paper's own 5%-noise results show that fingerprints of distinct model forms are not separated by more than the measurement noise in at least the Neo-Hooke case, which directly challenges the 'unique fingerprint' assumption on which the method rests. The reported training-data R2 values do not compensate for the absence of a confusion matrix or pairwise-distance analysis. The manuscript also currently lacks a coherent abstract matching its content.

major comments (4)
  1. [Section 4, Table 2 and Fig. 5] The central 'unique fingerprint' assumption is contradicted by the paper's own results. At 5% noise, the Neo-Hooke benchmark is identified as a Demiray model in the unsupervised setting (Table 2) and as an Ogden model with exponent 1.80 in the supervised setting. These are not parameter perturbations of the true model; they are different functional forms that happen to be closer to the noisy fingerprint than the true fingerprint is. The manuscript calls these 'alternative representations,' but this is exactly the failure mode that the uniqueness assumption excludes. To support the claim that nearest-neighbor lookup identifies material models, the paper needs a pairwise fingerprint-distance or confusion-matrix analysis showing that distinct models are separated by more than the noise level, with error bars or bounds. Currently this separation is asserted, not demonstrated.
  2. [Section 4, R2 reporting] The statement 'R2 exceeds 0.98 in all examples' is explicitly qualified as being computed on the training data. This metric only measures how well the discovered model fits the same noisy data used for matching; it does not measure whether the correct functional form was recovered. A wrong model with enough parameters can fit noisy training data. The paper should report held-out validation, e.g., predicting deformation states or loading paths not used in the fingerprint, and a model-identity confusion matrix, rather than relying on training-data R2 as evidence of successful identification.
  3. [Title and Abstract] The submitted manuscript's title and abstract describe an empirical study of large language models as expert annotators in finance, biomedicine, and law, but the full text is a computational mechanics paper on Material Fingerprinting. The abstract does not describe the paper's content, claims, or results in any way. This is not a minor typographical issue: as submitted, the paper lacks a valid abstract and would be misleading to readers and editors. The authors must correct the title/abstract or resubmit the correct manuscript before the technical content can be properly considered.
  4. [Section 4 and Appendix C] The clean-case exact recovery at 0% noise is near-constructive: the test fingerprints and the database entries are generated by the same finite element code, and the true model is always present in the database. This validates the implementation but does not establish discriminative power. The exclusion study in Appendix C acknowledges that a true model absent from the database cannot be recovered, and the method then returns an alternative model. This is a structural limitation of a lookup-table approach. The paper should discuss how the method can claim 'model discovery' when the model library must be specified a priori, and should report performance when the true model is not in the library or when the database grid is coarse relative to parameter separation.
minor comments (4)
  1. [Section 2.2.1] The variable 'ffi' appears repeatedly (e.g., 'we increase ffi during nt =10 equidistant load steps until reaching a maximum of ffi=0:3') and appears to be a PDF/LaTeX rendering artifact. It should be replaced with the actual symbol (presumably δ or ε) throughout the text and equations.
  2. [Tables 1 and 2] The columns Ecompr and Eincompr are not defined in the table captions or the surrounding text. Please define these error measures explicitly, including their normalization and units.
  3. [Section 4] The phrase 'ff =6 1:80' in the discussion of the Ogden model is garbled. It should read e.g., 'with exponent α ≠ 1.80' or similar. There are other encoding artifacts such as '⋋' and '⋌' in Table 2.
  4. [References] The reference 'Bensøe, 1989' appears to contain a typo; the correct spelling is likely 'Bendsøe'.

Circularity Check

2 steps flagged · score 6.0 of 10

Clean-case 'discovery' is a closed-loop lookup: the true model's fingerprint is generated by the same simulator used to build the database, and the reported R2 is computed on the training data; the paper's own 5% noise results further contradict the unique-fingerprint premise.

  1. self definitional [Section 2.2 and Section 4 (Tables 1/2)]
    "We then construct a material fingerprint database by simulating these experiments across various material models and parameters. ... At 0% noise, the discovered strain energy densities perfectly match the true models, yielding errors that vanish up to machine precision for each case."

    The database is populated by simulating the same candidate model families, and the benchmark data are generated with the same finite-element framework from one of those families. At zero noise the measured fingerprint coincides with the stored fingerprint of the generating model, so the nearest-neighbor lookup returns the truth by construction. This is a closed-loop consistency check, not an out-of-sample prediction; the answer is already an input to the database.

  2. fitted input called prediction [Section 4, Results (after Tables 1/2)]
    "The coefficient of determinationR 2, computed on the training data, exceeds 0.98 in all examples."

    R2 is evaluated on the very data that form the query and that determine the database entry chosen by nearest-neighbor matching. A high in-sample R2 is therefore forced by the selection rule and does not validate predictive ability on unseen states or specimens. Presenting this as evidence of effectiveness is a training-data metric renamed as a result, particularly because the same section shows a wrong model (Neo-Hooke→Demiray) can still achieve good agreement with noisy data.

full rationale

The paper's central idea—replace optimization with a precomputed fingerprint lookup—has independent algorithmic content, and the normalization/rescaling analysis goes beyond simple template matching. However, the benchmark evaluation is partially circular. At 0% noise, the test data are generated from a model family that is by design present in the database, so exact recovery is guaranteed up to numerical precision; this is retrieval of a stored answer, not discovery of an unknown form. The reported R2>0.98 is explicitly computed on the training data, so it cannot distinguish the true model from an alternative that fits the same noisy measurements. The paper's own 5% noise results exacerbate this: Neo-Hooke data are returned as Demiray (unsupervised) or Ogden (supervised), directly violating the stated 'unique fingerprint' premise and showing that the nearest-neighbor separation is not demonstrated. These are not self-citation issues; no load-bearing self-citation or imported uniqueness theorem appears. The circularity is therefore partial and concentrated in the evaluation logic rather than the method's internal consistency, warranting a score of 6 rather than 0-2.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim leans on two unpriced inputs: the curated model space of the database and the uniqueness of fingerprints. The paper states the uniqueness axiom but its own noisy results show non-unique identification (Neo-Hooke to Demiray/Ogden at 5% noise). The normalization insight is real, but its exact form is not in the provided text. No new physical entities are postulated.

free parameters (3)
  • Fingerprint database parameter grid (model forms and parameter ranges)
    The offline database contains a chosen sampling of material models and parameter values (referenced as Tables B.3 and B.4 in appendices not included in the provided text). The method's resolution and identifiability are determined by this grid; the paper notes the Blatz-Ko benchmark parameter (50.0) falls outside the database range yet is recovered via normalization.
  • Fingerprint normalization and parameter rescaling scheme
    Section 4 states normalization and rescaling are essential ('allows for accurately determining parameter magnitudes... a single fingerprint is sufficient to cover all possible parameter choices of models that depend on a single parameter'), but the exact normalization rules are not visible in the provided text. These choices control matching behavior and are not derived from data.
  • Standardized loading protocol (load steps, deformation amplitude, biaxial ratio) = n_t = 10 load steps, max ff = 0.3, 1:2 biaxial displacement ratio
    Section 2.2.1 fixes the standardized experiment used for both database generation and online testing. The discriminative power of the fingerprints depends on the probed deformation range; the values are design choices, stated but not optimized.
assumptions (3)
  • domain assumption Each material exhibits a unique response under a standardized experimental setup; the resulting fingerprint is a unique identifier encoding all pertinent mechanical information.
    Stated explicitly in the Abstract as 'the core assumption of Material Fingerprinting'. It is structurally separate from the method claim: the lookup fails if two distinct models produce indistinguishable fingerprints. Section 4 reports Neo-Hooke data identified as Demiray at 5% noise, a partial contradiction of uniqueness.
  • domain assumption The database contains a model form close enough to the true material behavior for a meaningful match to exist.
    Lookup can only return stored entries; functional-form 'discovery' is selection within the curated database. The paper demonstrates this in Appendix C, where excluding ground-truth models yields alternative models instead.
  • domain assumption The finite element simulation used to generate fingerprints faithfully represents the standardized experiment (geometry, boundary conditions, plane strain, displacement control).
    All fingerprints are simulations (Section 2.2.1). Any mismatch between the FE model and a real specimen enters directly into fingerprint space and corrupts matching; no experimental validation is provided.
invented entities (1)
  • material fingerprint
    purpose: Normalized response vector (stress measurements in the supervised setting; boundary reaction forces plus surface displacements in the unsupervised setting) used as a unique identifier for database matching.
    A methodological construct, not a physical entity: it adds no degree of freedom to the physics. Its usefulness rests entirely on the uniqueness assumption, which the paper's own 5% noise results partially refute.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Large Language Models as Expert Annotators." pith.science (2026). https://pith.science/paper/7SNY5KZC

@misc{pith2026250807827,
  author       = {Pith},
  title        = {Pith review of: Evaluating Large Language Models as Expert Annotators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7SNY5KZC}},
  note         = {Machine review of arXiv:2508.07827}
}
read the original abstract

Textual data annotation, the process of labeling or tagging text with relevant information, is typically costly, time-consuming, and labor-intensive. While large language models (LLMs) have demonstrated their potential as direct alternatives to human annotators for general domains natural language processing (NLP) tasks, their effectiveness on annotation tasks in domains requiring expert knowledge remains underexplored. In this paper, we investigate: whether top-performing LLMs, which might be perceived as having expert-level proficiency in academic and professional benchmarks, can serve as direct alternatives to human expert annotators? To this end, we evaluate both individual LLMs and multi-agent approaches across three highly specialized domains: finance, biomedicine, and law. Specifically, we propose a multi-agent discussion framework to simulate a group of human annotators, where LLMs are tasked to engage in discussions by considering others' annotations and justifications before finalizing their labels. Additionally, we incorporate reasoning models (e.g., o3-mini) to enable a more comprehensive comparison. Our empirical results reveal that: (1) Individual LLMs equipped with inference-time techniques (e.g., chain-of-thought (CoT), self-consistency) show only marginal or even negative performance gains, contrary to prior literature suggesting their broad effectiveness. (2) Overall, reasoning models do not demonstrate statistically significant improvements over non-reasoning models in most settings. This suggests that extended long CoT provides relatively limited benefits for data annotation in specialized domains. (3) Certain model behaviors emerge in the multi-agent discussion environment. For instance, Claude 3.7 Sonnet with thinking rarely changes its initial annotations, even when other agents provide correct annotations or valid reasoning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Agentic-imodels: Evolving agentic interpretability tools via autoresearch

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    Agentic-imodels evolves scikit-learn regressors via an autoresearch loop to jointly boost predictive performance and LLM-simulatability, improving downstream agentic data science tasks by up to 73% on the BLADE benchmark.

  2. Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotation

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    Double Triangle Annotation uses parallel MLLM consensus in two layers to reach WER 0.003 on 1887-1906 French medical directories while auto-accepting 85% of 13,595 fields via model agreement.

Reference graph

Works this paper leans on

52 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Open-source large language models outperform crowd workers and approach chatgpt in text-annotation tasks

    Meysam Alizadeh, Ma \"e l Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. Open-source large language models outperform crowd workers and approach chatgpt in text-annotation tasks. arXiv preprint arXiv:2307.02179, 2023

    arXiv 2023

  4. [4]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024

    2024

  5. [5]

    Claude 3.7 sonnet and claude code

    AI Anthropic. Claude 3.7 sonnet and claude code. 2025. URL https://www.anthropic.com/news/claude-3-7-sonnet

    work page 2025

  6. [6]

    Large language models as annotators: Enhancing generalization of nlp models at minimal cost

    Parikshit Bansal and Amit Sharma. Large language models as annotators: Enhancing generalization of nlp models at minimal cost. arXiv preprint arXiv:2306.15766, 2023

    arXiv 2023

  7. [7]

    Must read: A systematic survey of computational persuasion

    Nimet Beyza Bozdag, Shuhaib Mehri, Xiaocheng Yang, Hyeonjeong Ha, Zirui Cheng, Esin Durmus, Jiaxuan You, Heng Ji, Gokhan Tur, and Dilek Hakkani-T \"u r. Must read: A systematic survey of computational persuasion. arXiv preprint arXiv:2505.07775, 2025

    arXiv 2025

  8. [8]

    Can gpt models be financial analysts? an evaluation of chatgpt and gpt-4 on mock cfa exams

    Ethan Callanan, Amarachi Mbakwe, Antony Papadimitriou, Yulong Pei, Mathieu Sibue, Xiaodan Zhu, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. Can gpt models be financial analysts? an evaluation of chatgpt and gpt-4 on mock cfa exams. arXiv preprint arXiv:2310.08678, 2023

    arXiv 2023

Show all 52 references
  1. [9]

    Reconcile: Round-table conference improves reasoning via consensus among diverse llms

    Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309.13007, 2023

    2023 arXiv

  2. [10]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  3. [11]

    Cheng-Han Chiang and Hung-Yi Lee. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 15607--15631, 2023

    2023

  4. [12]

    Chatgpt goes to law school

    Jonathan H Choi, Kristin E Hickman, Amy B Monahan, and Daniel Schwarcz. Chatgpt goes to law school. J. Legal Educ., 71: 0 387, 2021

    2021

  5. [13]

    Gpts are multilingual annotators for sequence generation tasks

    Juhwan Choi, Eunju Lee, Kyohoon Jin, and YoungBin Kim. Gpts are multilingual annotators for sequence generation tasks. arXiv preprint arXiv:2402.05512, 2024

    2024 arXiv

  6. [14]

    Is gpt-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 11173--11195, 2023

    Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. Is gpt-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 11173--11195, 2023

    2023

  7. [15]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023

    2023 arXiv

  8. [16]

    Measuring the persuasiveness of language models, 2024

    Esin Durmus, Liane Lovitt, Alex Tamkin, Stuart Ritchie, Jack Clark, and Deep Ganguli. Measuring the persuasiveness of language models, 2024. URL https://www.anthropic.com/news/measuring-model-persuasiveness

    2024

  9. [17]

    Measuring nominal scale agreement among many raters

    Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76 0 (5): 0 378, 1971

    1971

  10. [18]

    Chatgpt outperforms crowd workers for text-annotation tasks

    Fabrizio Gilardi, Meysam Alizadeh, and Ma \"e l Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120 0 (30): 0 e2305016120, 2023

    2023

  11. [19]

    Introducing gemini 2.0: our new ai model for the agentic era

    Google. Introducing gemini 2.0: our new ai model for the agentic era. 2024. URL https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/#ceo-message

    2024

  12. [20]

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher R \'e , Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Ne...

    2024

  13. [21]

    Large language model based multi-agents: A survey of progress and challenges

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024

    2024 arXiv

  14. [22]

    Annollm: Making large language models to be better crowdsourced annotators

    Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854, 2023

    2023 arXiv

  15. [23]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020

    2020

  16. [24]

    Cuad: An expert-annotated nlp dataset for legal contract review

    Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268, 2021

    2021 arXiv

  17. [25]

    Coda-19: Using a non-expert crowd to annotate research aspects on 10,000+ abstracts in the covid-19 open research dataset

    Ting-Hao Kenneth Huang, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Yen-Chia Hsu, and C Lee Giles. Coda-19: Using a non-expert crowd to annotate research aspects on 10,000+ abstracts in the covid-19 open research dataset. In ACL 2020 Workshop on Natural Language Processing fo...

    2020

  18. [26]

    Pubmedqa: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natur...

    2019

  19. [27]

    Gpt-4 passes the bar exam

    Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382 0 (2270): 0 20230254, 2024

    2024

  20. [28]

    Refind: Relation extraction financial dataset

    Simerjot Kaur, Charese Smiley, Akshat Gupta, Joy Sain, Dongsheng Wang, Suchetha Siddagangappa, Toyin Aguda, and Sameena Shah. Refind: Relation extraction financial dataset. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information...

    2023

  21. [29]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

    2022

  22. [30]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023

    2023 arXiv

  23. [31]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    Note on the sampling error of the difference between correlated proportions or percentages

    Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12 0 (2): 0 153--157, 1947

    1947

  25. [33]

    Hello gpt4-o

    OpenAI. Hello gpt4-o. 2024. URL https://openai.com/index/hello-gpt-4o/

    2024

  26. [34]

    Openai o3-mini

    OpenAI. Openai o3-mini. 2025. URL https://openai.com/index/openai-o3-mini/

    2025

  27. [35]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv pre...

    2024 arXiv

  28. [36]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  29. [37]

    Trillion dollar words: A new financial dataset, task & market analysis

    Agam Shah, Suvan Paturi, and Sudheer Chava. Trillion dollar words: A new financial dataset, task & market analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 6664--6679, 2023

    2023

  30. [38]

    Large language models encode clinical knowledge

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620 0 (7972): 0 172--180, 2023 a

    2023

  31. [39]

    Towards expert-level medical question answering with large language models

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617, 2023 b

    2023 arXiv

  32. [40]

    Large language models for data annotation: A survey

    Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation: A survey. arXiv preprint arXiv:2402.13446, 2024

    2024 arXiv

  33. [41]

    Are expert-level language models expert-level annotators? arXiv preprint arXiv:2410.03254, 2024 a

    Yu-Min Tseng, Wei-Lin Chen, Chung-Chi Chen, and Hsin-Hsi Chen. Are expert-level language models expert-level annotators? arXiv preprint arXiv:2410.03254, 2024 a

    2024 arXiv

  34. [42]

    Two tales of persona in llms: A survey of role-playing and personalization

    Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Yu-Ching Hsu, Jia-Yin Foo, Chao-Wei Huang, and Yun-Nung Chen. Two tales of persona in llms: A survey of role-playing and personalization. arXiv preprint arXiv:2406.01171, 2024 b

    2024 arXiv

  35. [43]

    Foundational autoraters: Taming large language models for better automatic evaluation

    Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 1708...

    2024

  36. [44]

    Cord-19: The covid-19 open research dataset

    Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Douglas Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, et al. Cord-19: The covid-19 open research dataset. ArXiv, 2020

    2020

  37. [45]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022

    2022

  38. [46]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  39. [47]

    The rise and potential of large language model based agents: A survey

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68 0 (2): 0 121101, 2025

    2025

  40. [48]

    Llmaaa: Making large language models as active annotators

    Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. Llmaaa: Making large language models as active annotators. arXiv preprint arXiv:2310.19596, 2023

    2023 arXiv

  41. [49]

    Can chatgpt reproduce human-generated labels? a study of social computing tasks

    Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce human-generated labels? a study of social computing tasks. arXiv preprint arXiv:2304.10145, 2023

    2023 arXiv

  42. [50]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  43. [51]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  44. [52]

    By leveraging additional inference-time compute, we explore whether individual LLMs can serve as a direct alternative to expert data annotators

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.