Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This position paper argues that descriptive datasheets cannot measure dataset quality and proposes DataRubrics, a ten-dimension rubric with LLM-as-a-judge evaluation, as the path to reproducible, scalable dataset review.

desk verdict Useful rubric proposal, but the 26% human-error headline is built on a circular LLM-as-judge validation. read the letter →

arxiv 2506.01789 v2 pith:F6GNC4ZF submitted 2025-06-02 cs.LG cs.AIcs.CLcs.CVeess.AS

Genta Indra Winata , David Anugraha , Emmy Liu , Alham Fikri Aji , Shou-Yi Hung , Aditya Parashar , Patrick Amadeus Irawan , Ruochen Zhang
show 12 more authors
Zheng-Xin Yong Jan Christian Blaise Cruz Niklas Muennighoff Seungone Kim Hanyang Zhao Sudipta Kar Kezia Erina Suryoraharjo M. Farid Adilazuarda En-Shiun Annie Lee Ayu Purwarianti Derry Tanti Wijaya Monojit Choudhury
This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CVeess.AS
keywords datasetqualitydatasheetsrubricsLLM-as-a-judgereviewpeerreproducibilitydata-centricAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dataset papers are hard to review because datasheets and checklists describe a dataset but never measure its quality, and conference metadata rules are inconsistently enforced. The paper argues that dataset review needs rubric-based, quantifiable evaluation, and proposes DataRubrics, a ten-dimension framework covering data sources, annotators, novelty, creation process, task utility, annotation guidelines, quality assurance, human and non-human language coverage, and code availability, with each judgment backed by a reference to the paper being reviewed. To demonstrate feasibility, the authors manually scored 100 NeurIPS dataset-track papers and used an LLM judge to score papers from seven conferences over 2021–2024, producing trend statistics. Their central evidence for automation is that human annotations still contained 26% errors after quality assurance, and that model-assisted annotation could catch such errors. A sympathetic reading is that the structured schema, not the particular LLM, is the core contribution.

What carries the argument

The load-bearing machinery is the DataRubrics schema itself: a JSON-structured scoring rubric, one template per dimension, that forces the evaluator (human or LLM) to output multi-label booleans plus a locator reference (for example, 'Section 3.1') and free-text reasoning for each label. The scalable pipeline around it is: OCR extracts clean text from paper PDFs, a reward model filters papers whose title and abstract introduce a dataset or benchmark, and an LLM judge is prompted with the dimension template under structured decoding to produce the labels. The reference-and-reasoning requirement is what distinguishes DataRubrics from binary checklists: every assertion is traceable to a specific place in the paper.

What would settle it

Take the same 100 NeurIPS dataset-track papers and have three expert annotators independently label them with adjudicated consensus; if the LLM judge's labels disagree with that gold standard at a rate comparable to or larger than the 26% it attributes to human annotators, the claim that model-assisted annotation reduces human error fails. A simpler decisive check would be to release the exact prompt, the OCR text, and the model outputs for one conference so an independent team can reproduce the 26% figure verbatim.

Watch

Extended reading notes

Core claim

The central claim, offered as a position and supported by a pipeline demonstration, is that evaluative dataset quality can be operationalized as a fixed ten-dimension rubric with multi-label answers, each answer accompanied by a quoted reference to the paper under review and a short reasoning sentence. The same schema is readable by human annotators and by an LLM judge under structured decoding, so the authors can generate conference-level trend statistics automatically. On their evidence, the share of papers proposing model-generated data is rising steadily across venues, CVPR shows the lowest adoption of quality documentation, and human annotation of papers is error-prone enough after quality assurance that model-assisted review should be part of the pipeline. The claim is not that any single label is always correct, but that a structured schema makes disagreements visible, verifiable, and correctable.

Load-bearing premise

The framework's usefulness rests on the premise that an LLM judge reading OCR-extracted paper text gives rubric labels accurate enough to stand in for careful human review, and the paper's validation against human annotators does not include an independent gold standard for deciding which side is right.

Editorial extensions

If this is right

  • If DataRubrics is adopted, dataset review can move from open-ended datasheets to a common ten-dimension scorecard, letting reviewers compare papers across venues and years on the same axes.
  • Conference organizers could pre-compute rubric labels for each submission and hand reviewers a section-referenced quality summary, easing the load from rising submission volume.
  • The measured rise in model-generated data across all conferences implies that automated quality assessment becomes more necessary, not less, as synthetic data grows.
  • The 26% residual human-annotation error rate, if accurate, implies that a single human QA pass on dataset papers is insufficient and that model-assisted double-checking is warranted.
  • By tracking non-human language coverage (code, mathematical notation, biological sequences), DataRubrics extends to dataset types that existing datasheets and checklists ignore.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the rubric labels prove reliable beyond this study, the same ten-dimension schema could be applied at runtime to the dataset content itself, giving users a quality badge before they download or use the data.
  • The 26% figure is a comparison between one LLM and human annotators, not against an independent gold standard; a fair reading suggests the honest conclusion is that the two disagree, and resolving that disagreement is the next test rather than proof of a fixed human failure rate.
  • The same schema could be repurposed as an author-side self-check at submission time, telling authors which quality dimensions their paper fails to document before referees see it.
  • If structured-decoding outputs are cheap enough, the method could run on every dataset paper in a venue, turning the rubric into a continuously updated leaderboard instead of a per-submission review tool.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This position paper argues that existing datasheets and checklists are inadequate for evaluating dataset quality and proposes DataRubrics, a ten-dimension rubric framework with structured multi-label schemas designed for both human assessment and LLM-as-a-judge evaluation. The authors survey dataset-related policies at major conferences, collect and OCR-filter papers from several venues, automatically annotate them with GPT-4.1-mini, manually annotate 100 NeurIPS Datasets and Benchmarks papers with expert QA, and compare human and LLM annotations. They report trends across conferences and claim that 26% of QA-passed human annotations are incorrect, concluding with recommendations and a code release.

Significance. The proposal is timely and the rubric design is a useful contribution; the structured schemas and the released code are concrete assets that could lower the cost of dataset quality assessment. However, the empirical validation is currently circular and incomplete: the LLM judge is used as the reference standard for the human-error estimate, no independent gold standard or agreement statistics are provided, and all trend analyses inherit the unvalidated judge's potential biases. If the empirical claims were properly validated or appropriately weakened, the framework could be a valuable tool for dataset review, but in its present form the evidence does not support the headline claims.

major comments (3)
  1. [§5.4.2, Figure 7] The claim that 26% of QA-passed human annotations are incorrect is circular: GPT-4.1-mini serves as both the automatic evaluator and the reference standard for what is 'correct.' No independent gold standard (e.g., adjudicated expert labels) is provided, and no inter-annotator agreement statistics (e.g., Cohen's kappa) are reported for either the human experts or the LLM. Because the paper itself concedes in the same paragraph that similar misclassification patterns are observed in automatic metrics, the 26% figure reflects disagreement between one LLM judge and the human annotators, not a measured human error rate. The authors should validate the LLM labels against adjudicated human gold labels on a sample and report a disagreement decomposition (LLM-only errors, human-only errors, and joint errors) rather than labeling all disagreements as human errors.
  2. [§5.4.1 and Appendix A] All conference-level trends (e.g., CVPR having the lowest quality-assurance reporting, the steady increase in model-generated data) are computed exclusively from GPT-4.1-mini labels, with no validation of the judge's accuracy on this task, no error bars or confidence intervals, and no release of the random sampling seed described in §5.1. With a sampling cap of 100 papers per conference-year and no reported seed or selection protocol, the trends cannot be reproduced and may reflect sampling variance or systematic LLM bias. Please report the exact sampling methodology (seeds, inclusion/exclusion criteria, number of papers actually analyzed per conference-year cell) and provide uncertainty estimates or a human-validated subset to support the trend claims.
  3. [§5.2 and §5.3] The description of the evaluation pipeline omits key details required to assess reliability: the exact version of GPT-4.1-mini, decoding parameters (temperature, number of runs, any ensembling), and the treatment of OCR errors introduced by OlmOCR. Additionally, §5.3 assigns each paper to a single expert followed by a second-expert QA step, but no agreement statistics between the two experts are reported, so the reliability of the human annotations themselves is unquantified. Please add these methodological details and, if feasible, a small error analysis of how OCR extraction failures affect downstream label accuracy.
minor comments (5)
  1. [Title] The title contains a typo: 'DATARUBRICSfor' should be 'DATARUBRICS for'.
  2. [§5.4.2] The phrase 'reannotate a set of 100 data points' should be '100 papers' for clarity, and the model name is inconsistently written as both 'GPT-4.1-mini' and 'GPT-4.1 mini' across the text.
  3. [Appendix B] The schemas contain several typos and inconsistencies: the Quality Assurance schema requires 'Automatic Verification' in one list but defines 'Automatic Process' in its properties (§B.3.2); the Language Coverage schema is named 'Task Utility' in §B.6.2; and there are typos such as 'appliest' and 'langauges'.
  4. [Figure 2 caption] The caption states that 'only the ICLR 2025 proceedings had been published' at the time of writing, but the data collection section says the study period is 2021-2024; please clarify whether 2025 data is included or excluded in the trend analyses to avoid ambiguity.
  5. [§3.1] The claim that 'anecdotal stories have emerged of papers being rejected due to having too few languages or small dataset sizes' is unsupported; please provide citations or remove the anecdotal framing.

Circularity Check

1 steps flagged · score 4.0 of 10

The 26% human-error claim in §5.4.2 is circular: GPT-4.1-mini labels are used as the reference standard to declare human annotations 'incorrect,' so the statistic measures human–LLM disagreement rather than independently validated human error.

  1. self definitional [Section 5.4.2 (Automatic Evaluation vs. Human Evaluation), Figure 7]
    "We find that human annotations still contain errors even after undergoing quality assurance (QA) procedures. To better understand the nature of these errors, we reannotate a set of 100 data points using outputs from GPT-4.1 mini, focusing on NeurIPS papers. As shown in Figure 7, 26% of the annotations remain incorrect, despite having passed QA by human annotators tasked with identifying whether a paper includes specific annotations."

    The paper uses GPT-4.1-mini—the same LLM-as-a-judge system whose outputs drive all automatic evaluations in §5.4.1 and Appendix A—as the reference standard for labeling human annotations 'incorrect.' No independent gold standard or adjudicated ground truth is provided, so 'incorrect' is defined by construction as 'disagrees with GPT-4.1-mini.' The 26% figure is therefore a human–LLM disagreement rate, not a measured human error rate. The subsequent conclusion that 'model-assisted annotation could play a valuable role in reducing human error' assumes the LLM's labels are correct, which is precisely what the comparison fails to establish.

full rationale

The DataRubrics framework itself is a design proposal with independently stated dimensions and rubric templates in §4 and Appendix B; it is not mathematically derived from the empirical results, so the core proposal retains non-circular content. The main circularity is localized to §5.4.2, where GPT-4.1-mini outputs are used as the ground truth to declare 26% of QA-passed human annotations 'incorrect.' This is a self-definitional comparison: the error statistic reduces by construction to human–LLM disagreement. The trend analyses in §5.4.1 and Appendix A, while unvalidated against an external gold standard, are descriptive of the LLM labels and are not themselves circular. The cited reward model R3 [4] and MetaMetrics [21] are used for context and filtering, not as load-bearing uniqueness theorems or as the sole justification of the paper's central proposal. Because one empirical claim reduces by construction but the central framework retains independent content, the appropriate score is 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The framework rests on hand-selected dimensions and label sets, and its evaluation pipeline assumes LLM and OCR reliability. No numerical constants are fitted, but the design choices above function as free parameters because they fully determine the outputs.

free parameters (3)
  • Ten rubric dimensions = fixed set of 10
    The dimensions in Section 4.1 are author-selected and not derived from an external standard; they determine the scope of what the framework can measure.
  • Per-dimension label sets = hand-built enums (e.g., annotator categories, utility labels, language labels)
    The JSON enums in Appendix B constrain LLM outputs; they are design choices, not empirically justified categories.
  • Sampling cap of 100 papers per conference-year = 100
    Section 5.1 caps the analysis at 100 randomly sampled papers per year per conference without seeds or paper lists; this choice directly shapes the trend figures.
assumptions (4)
  • domain assumption LLM-as-a-judge reliability
    The framework assumes a structured-prompt LLM labels dataset papers accurately enough to rank or assess quality, as used in Section 5.2.
  • domain assumption OCR extraction fidelity
    OlmOCR is assumed to extract paper text completely and faithfully for evaluation, as described in Section 5.1.
  • domain assumption Reward-model filtering correctness
    R3-Qwen3-14B-4k is assumed to identify dataset and benchmark papers from title and abstract without systematic bias, as used in Section 5.1.
  • domain assumption Human annotation reliability
    Section 5.3 treats single-expert annotations plus one QA pass as a usable reference, yet Section 5.4.2 reports 26% residual errors; the paper does not resolve this tension.
invented entities (1)
  • DataRubrics framework
    purpose: Structured rubric schema for evaluating dataset quality across ten dimensions in human and machine-readable form
    The framework is new, but the paper provides no external benchmark showing its labels correlate with actual dataset quality; the pilot validation uses the same LLM-as-a-judge paradigm it proposes, so independent evidence is absent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability." pith.science (2026). https://pith.science/paper/F6GNC4ZF

@misc{pith2026250601789,
  author       = {Pith},
  title        = {Pith review of: Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F6GNC4ZF}},
  note         = {Machine review of arXiv:2506.01789}
}
read the original abstract

High-quality datasets are fundamental to training and evaluating machine learning models, yet their creation-especially with accurate human annotations-remains a significant challenge. Many dataset paper submissions lack originality, diversity, or rigorous quality control, and these shortcomings are often overlooked during peer review. Submissions also frequently omit essential details about dataset construction and properties. While existing tools such as datasheets aim to promote transparency, they are largely descriptive and do not provide standardized, measurable methods for evaluating data quality. Similarly, metadata requirements at conferences promote accountability but are inconsistently enforced. To address these limitations, this position paper advocates for the integration of systematic, rubric-based evaluation metrics into the dataset review process-particularly as submission volumes continue to grow. We also explore scalable, cost-effective methods for synthetic data generation, including dedicated tools and LLM-as-a-judge approaches, to support more efficient evaluation. As a call to action, we introduce DataRubrics, a structured framework for assessing the quality of both human- and model-generated datasets. Leveraging recent advances in LLM-based evaluation, DataRubrics offers a reproducible, scalable, and actionable solution for dataset quality assessment, enabling both authors and reviewers to uphold higher standards in data-centric research. We also release code to support reproducibility of LLM-based evaluations at https://github.com/datarubrics/datarubrics.

Figures

Figures reproduced from arXiv: 2506.01789 by the authors.

Figure 1
Figure 1. We study the datasheets and checklists in 10 dimensions of data quality. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of yearly trends: all accepted papers [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Inference pipeline using DataRubrics. We use OCR to extract text from PDFs, filter [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Paper statistics across conferences. Only the 2022 and 2024 editions of LREC are included, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Percentage of papers on NeurIPS with Guidelines on their data collection. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Percentage of papers with proposed new data generated from model. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Comparison of human–LLM annotation agreement. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Paper statistics across conferences on Data Annotations at ACL, CVPR, EMNLP, and ICLR [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Paper statistics across conferences on Data Annotations at ICML, Interspeech, LREC, and [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Paper statistics across conferences on Data Novelty at ACL, CVPR, EMNLP, and ICLR [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Paper statistics across conferences on Data Novelty at ICML, Interspeech, LREC, and [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Paper statistics across conferences on Languages at ACL, CVPR, EMNLP, and ICLR [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Paper statistics across conferences on Languages at ICML, Interspeech, LREC, and [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Paper statistics across conferences on Quality Assurance at ACL, CVPR, EMNLP, and [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Paper statistics across conferences on Quality Assurance at ICML, Interspeech, LREC, [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Paper statistics across conferences on Data Creation and Code at ACL, CVPR, EMNLP, [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Paper statistics across conferences on Data Creation and Code at ICML, Interspeech, [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Paper statistics across conferences on Task Utility at ACL, CVPR, EMNLP, and ICLR [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Paper statistics across conferences on Task Utility at ICML, Interspeech, LREC, and [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 13 canonical work pages

  1. [1]

    Akhtar, O

    M. Akhtar, O. Benjelloun, C. Conforti, L. Foschini, J. Giner-Miguelez, P. Gijsbers, S. Goswami, N. Jain, M. Karamousadakis, M. Kuchnik, et al. Croissant: A metadata format for ml-ready datasets.Advances in Neural Information Processing Systems, 37:82133–82148, 2024

    work page 2024

  2. [2]

    Ankner, M

    Z. Ankner, M. Paul, B. Cui, J. D. Chang, and P. Ammanabrolu. Critique-out-loud reward models.arXiv preprint arXiv:2408.11791, 2024

    arXiv 2024

  3. [3]

    ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models

    D. Anugraha, G. I. Winata, C. Li, P. A. Irawan, and E.-S. A. Lee. Proxylm: Predicting language model performance on multilingual tasks via proxy models.arXiv preprint arXiv:2406.09334, 2024

    work page Pith review arXiv 2024

  4. [4]

    Anugraha, Z

    D. Anugraha, Z. Tang, L. J. V . Miranda, H. Zhao, M. R. Farhansyah, G. Kuwanto, D. Wijaya, and G. I. Winata. R3: Robust rubric-agnostic reward models.arXiv preprint arXiv:2505.13388, 2025

    arXiv 2025

  5. [5]

    E. M. Bender and B. Friedman. Data statements for natural language processing: Toward mitigat- ing system bias and enabling better science.Transactions of the Association for Computational Linguistics, 6:587–604, 2018

    work page 2018

  6. [6]

    X. Chen, G. Li, Z. Wang, B. Jin, C. Qian, Y . Wang, H. Wang, Y . Zhang, D. Zhang, T. Zhang, et al. Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387, 2025

    arXiv 2025

  7. [7]

    M. R. Farhansyah, I. Darmawan, A. Kusumawardhana, G. I. Winata, A. F. Aji, and D. T. Wijaya. Do language models understand honorific systems in javanese?arXiv preprint arXiv:2502.20864, 2025

    work page Pith review arXiv 2025

  8. [8]

    Gebru, J

    T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford. Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    2021

Show all 22 references
  1. [9]

    Holland, A

    S. Holland, A. Hosny, S. Newman, J. Joseph, and K. Chmielinski. The dataset nutrition label: A framework to drive higher data quality standards.arXiv preprint arXiv:1805.03677, 2018

    2018 arXiv

  2. [10]

    Hutchinson, A

    B. Hutchinson, A. Smart, A. Hanna, E. Denton, C. Greer, O. Kjartansson, P. Barnes, and M. Mitchell. Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. InProceedings of the 2021 ACM conference on fairness, accountabilit...

    2021

  3. [11]

    Mihalcea, O

    R. Mihalcea, O. Ignat, L. Bai, A. Borah, L. Chiruzzo, Z. Jin, C. Kwizera, J. Nwatu, S. Poria, and T. Solorio. Why ai is weird and shouldn’t be this way: Towards ai for everyone, with everyone, by everyone. InProceedings of the AAAI Conference on Artificial Intelligence, pages ...

    2025

  4. [12]

    Naous, M

    T. Naous, M. J. Ryan, A. Ritter, and W. Xu. Having beer after prayer? measuring cultural bias in large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, 2024

    2024

  5. [13]

    A. H. Nasution and A. Onan. Chatgpt label: Comparing the quality of human-generated and llm-generated annotations in low-resource language nlp tasks.IEEE Access, 12:71876–71900, 2024

    2024

  6. [14]

    Poznanski, J

    J. Poznanski, J. Borchardt, J. Dunkelberger, R. Huff, D. Lin, A. Rangapur, C. Wilhelm, K. Lo, and L. Soldaini. olmocr: Unlocking trillions of tokens in pdfs with vision language models. arXiv preprint arXiv:2502.18443, 2025. 11

    2025

  7. [15]

    R. A. Putri, F. G. Haznitrama, D. Adhista, and A. Oh. Can llm generate culturally relevant com- monsense qa data? case study in indonesian and sundanese.arXiv preprint arXiv:2402.17302, 2024

    2024 arXiv

  8. [16]

    Reuel, A

    A. Reuel, A. Hardy, C. Smith, M. Lamparth, M. Hardy, and M. J. Kochenderfer. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices.arXiv preprint arXiv:2411.12990, 2024

    2024 arXiv

  9. [17]

    Shumailov, Z

    I. Shumailov, Z. Shumaylov, Y . Zhao, et al. Ai models collapse when trained on recursively generated data.Nature, 631:755–759, 2024. doi: 10.1038/s41586-024-07566-y

    2024 doi

  10. [18]

    Veselovsky, M

    V . Veselovsky, M. H. Ribeiro, and R. West. Artificial artificial artificial intelligence: Crowd work- ers widely use large language models for text production tasks.arXiv preprint arXiv:2306.07899, 2023

    2023 arXiv

  11. [19]

    T. Vu, K. Krishna, S. Alzubi, C. Tar, M. Faruqui, and Y .-H. Sung. Foundational autoraters: Taming large language models for better automatic evaluation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17086–17105, 2024

    2024

  12. [20]

    X. Wang, H. Kim, S. Rahman, K. Mitra, and Z. Miao. Human-llm collaborative annotation through effective verification of llm labels. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–21, 2024

    2024

  13. [21]

    G. I. Winata, D. Anugraha, L. Susanto, G. Kuwanto, and D. T. Wijaya. Metametrics: Cali- brating metrics for generation tasks using human preferences. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=slO3xTt4CG

    2025

  14. [22]

    name": "sources

    M. Wu and A. F. Aji. Style over substance: Evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 297–312, 2025. 12 A Paper Statistics via Automatic Evaluation We present paper trends from 2022 to 20...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.