Pith. sign in

REVIEW 3 major objections 6 minor 62 references

This paper proposes that idea-level similarity between LLM and expert annotations can be benchmarked with a pick-the-odd-one-out task, and reports that LLM judges prompted in that format beat lexical and vector baselines by 9-30% on two edu

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

IDEAlign uses a pick-the-odd-one-out triplet task to measure idea-level similarity between LLMs and expert human annotations, and shows LLM judges using this protocol outperform lexical and vector-based baselines.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A transparent benchmark with a credible central result, but the headline number is loose and the pooled Spearman needs a within-source check. the 3 major comments →

arxiv 2509.02855 v1 pith:7MSW4WLA submitted 2025-09-02 cs.CL cs.CY

IDEAlign: Comparing Large Language Models to Human Experts in Open-ended Interpretive Annotations

Hyunji Nam , Lucia Langlois , James Malamut , Mei Tan , Dorottya Demszky This is my paper
classification cs.CL cs.CY
keywords IDEAlignpick-the-odd-one-outLLM-as-a-judgeinterpretive annotationsimilarity benchmarkingtext embeddingstopic modelseducational feedback
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-ended annotation tasks—a teacher commenting on a student essay, a researcher reading a classroom transcript—have no single correct answer, so checking whether an LLM shares an expert's ideas is hard to do and hard to scale. The paper introduces IDEAlign, a benchmarking protocol that replaces absolute similarity ratings with a simple relative judgment: given three annotations, pick the most different one. Aggregating many such triplet decisions yields pairwise similarity scores, and the same protocol can be posed to human experts and to LLM judges. Using expert labels on two educational tasks (assessing students' math reasoning and giving essay feedback), the paper finds that embedding- and topic-based similarity metrics correlate weakly with experts, while LLM judges prompted with the triplet format reach Spearman correlations of 0.618 and 0.683—a 9-30% improvement over the best lexical and vector baselines. If the result holds, the paper offers a practical way to measure idea-level agreement between machines and experts at scale, and a warning that standard vector metrics are measuring something closer to style than substance.

Core claim

The claim is that idea-level alignment between LLM and expert annotations can be measured reliably by aggregating 'pick-the-odd-one-out' judgments, and that LLM judges evaluate similarity far more like experts when prompted with that same triplet protocol than when they score pairs directly or when text embeddings or topic models are used. The paper reports Spearman rank correlations of 0.618 (feedback) and 0.683 (reasoning) between the best LLM triplet judge and the expert benchmark, versus at best near 0.49 for vector-based baselines; it also shows the LLM-judge ranking is far less affected by a controlled style change that inflated embedding similarities by up to 82%. The authors conclude

What carries the argument

The central mechanism is the triplet judgment task and its aggregation formula. In IDEAlign, a judge sees three annotations of the same source and picks the one most different under a domain-specific criterion; the similarity of a pair A,B is then the fraction of triplets containing both in which neither A nor B was picked as the odd one out (Equation 1). Because the identical protocol is used for expert humans and for LLM-as-a-judge, the two can be compared on equal footing, and relative judgments avoid the calibration noise of absolute rating scales.

Load-bearing premise

The load-bearing premise is that similarity scores computed from triplets within different source documents land on one comparable scale, so all pairs can be ranked together and summarized by a single correlation; if triplet composition differs per document, the pooled ranking may not be commensurable.

What would settle it

Recompute the Spearman correlations separately within each transcript or essay instead of pooling all pairs. If the within-document correlations for the LLM triplet judge fall near zero or vary wildly across documents while the pooled values sit at 0.618/0.683, the reported alignment is an artifact of pooling incomparable scores.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Expert similarity ground truth can be collected with a cognitively easy task: multiple humans pick the odd one out of three annotations, and aggregation produces continuous pairwise scores.
  • Once validated against experts in a domain, an LLM prompted with the triplet format can serve as a scalable proxy for idea-level similarity.
  • Embedding- and topic-based similarity metrics should not be trusted as proxies for expert judgments on open-ended annotations; the style-perturbation test shows their scores can move 20-82% when only wording changes.
  • Dataset size matters: bootstrapping the expert judgments suggests roughly 300 triplets suffice for the feedback task and about 600 for the reasoning task before rankings stabilize, a check practitioners should run.
  • Adding domain metadata to the prompt did not reliably help LLM judges and often hurt, so the gap between LLM and expert use of domain knowledge remains open.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pattern generalizes, the same triplet protocol could be ported to other interpretive domains (clinical notes, policy analysis, peer review) where the question is whether two texts express the same idea rather than share surface form.
  • The results suggest a training signal: IDEAlign scores could be used to fine-tune embedding models to discount style and length, turning the paper's negative embedding results into a positive optimization target.
  • A direct testable extension would apply IDEAlign to a task where experts intentionally disagree about the similarity criterion; the paper's instruction-led protocol predicts that triplet aggregation would still produce stable rankings, while absolute ratings would not.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces IDEAlign, a benchmark protocol in which expert raters see triplets of open-ended annotations from the same source and choose the odd one out along a substantive dimension; Eq. (1) converts these decisions into pairwise similarity proportions. The authors collect such judgments for two education tasks (math-reasoning annotations and essay feedback), then evaluate BLEU, 16 embedding models with and without PCA post-processing, BERTopic/LDA topic models, and LLM-as-a-judge in binary, continuous, and triplet formats. Using pooled Spearman rank correlations, they claim GPT-4.1 triplet judgments outperform lexical and vector-based baselines by 9-30%, and they include supporting analyses of metadata sensitivity, style perturbation, and bootstrap-based sample-size adequacy.

Significance. If the central empirical claim survives scrutiny, IDEAlign is a useful and low-cost protocol: relative judgments reduce the calibration noise of absolute ratings, and the same triplet prompt can be administered to both human experts and LLMs. The paper evaluates on real educational data with paid experts, compares many metrics (16 embeddings, topic models, 4 LLMs), and provides code/data links. The oracle bootstrap correlations (0.906 and 0.812) indicate that the human labels are informative. I do not see a circularity problem: the human judgments are an external gold standard, and no metric is fit to them. However, the headline quantitative claim is currently impaired by an unexamined cross-source pooling issue and by an inconsistent baseline used for the 9-30% range; these are fixable with additional analyses.

major comments (3)
  1. [Section 4.5, Eq. (1)] The headline correlations pool pairwise scores across all source documents. sim(A,B) is a proportion of triplet judgments in which neither A nor B was selected; the denominator and the composition of alternatives are source-specific, so the scale is not calibrated across transcripts/essays. A score of 0.8 in one source need not mean the same judged similarity as 0.8 in another. Ranking all pairs together and computing one Spearman rho therefore mixes within-source ordering with potentially arbitrary between-source differences, and the reported 9-30% LLM advantage could be driven by ecological source-level alignment. Please report within-source rho values (averaged or stratified) for Tables 2, 3, 6-9 and Figure 4, or otherwise justify that the Eq. (1) scale is commensurable across sources.
  2. [Abstract and Section 6.2] The '9-30% increase' is not reproducible from the reported tables. Using the best baselines in Table 2 (MiniLM-L6, 0.489 on feedback; BERTopic, 0.491 on reasoning), GPT-4.1 triplet's 0.618/0.683 correspond to 26% and 39% increases, not 9-30%. Even if one takes the best Appendix baselines (cpt-text-large 0.563 for feedback, BERTopic min_cluster=9 at 0.518 for reasoning), the improvements are about 10% and 32%. Moreover, Section 6.2 attributes the 30% improvement to 'the best-performing text embedding-based metrics,' which cannot be true for reasoning since the best embedding there is 0.212 after PCA. Please state explicitly which baselines define the range and recompute the abstract claim, or revise the claim.
  3. [Section 6.2 and Discussion] The paper repeatedly uses 'significantly improves' without reporting uncertainty or significance tests for the Spearman differences. The only bootstrap analysis is for the oracle human-human correlation, not for the comparison between GPT-4.1 triplet judgments and the baselines. With a modest number of pairs, 0.618 versus 0.489 or 0.683 versus 0.491 could be within sampling noise. Add a bootstrap or permutation procedure over pairs/triplets for Tables 2 and 3, or soften the wording to 'numerically higher' until significance is established.
minor comments (6)
  1. [Abstract] The abbreviation is written as 'IDEAlgin' in the abstract but 'IDEAlign' in the title and body; please make it consistent.
  2. [Section 6.3, Table 4] The text says 'longer contexts actually hurt performance in all models except Sonnet 3.5,' but Table 4 shows Sonnet 4 reasoning improving by 16.98% with metadata. Please correct the sentence or the table.
  3. [Section 6.5] The sample-size analysis is described at a high level. It is unclear whether bootstrap resamples triplet judgments, pairs, or entire source documents, and why the reasoning data is said to require '600' tasks when Section 5.1 reports 640 triplet comparisons. Clarify the unit of resampling and reconcile the counts.
  4. [Appendix E] The style perturbation rewrites appear to change content as well as style. For example, 'That 100 is 10' becomes '10 to the power of 0 is 10,' and 'counting 10s by 1's, 2's and 3's' becomes 'counting the powers of 10.' The conclusion that 'only style' was changed is therefore not supported; please restrict the rewrites to surface-level form or revisit the interpretation.
  5. [Section 4.4] Please specify the LLM sampling temperature, whether the 10 samples are drawn from the same prompt or with different formatting, and explicitly confirm that the LLMs were evaluated on the same triplet set as the human experts. Otherwise, the aggregation in Eq. (1) may not be exactly comparable across human and LLM judgments.
  6. [Appendix A.3, Table 9] The topic-modeling hyperparameters are selected by maximizing correlation with the human labels on the same data. This is explained in the caption but should also be stated in the main text; otherwise readers may mistake the best hyperparameter result for a pre-registered or fixed configuration.

Circularity Check

0 steps flagged

No significant circularity: human expert judgments are an external benchmark, and the LLM-as-a-judge scores are evaluated against them without fitting any parameters to those labels.

full rationale

The paper's central claim is that LLM-as-a-judge scores, especially under the triplet protocol, align with expert human similarity judgments. The human judgments are collected independently from recruited expert educators (Sections 5.1 and 5.2) and aggregated using the transparent formula in Equation (1). The LLM scores are produced with the same triplet task format but are not fitted, calibrated, or otherwise tuned to the human labels; the Spearman correlations in Section 4.5 are computed between two separate rankings after the fact. No parameter of the LLM judge is optimized against the benchmark. The only symmetry is that humans and LLMs perform the same odd-one-out task, which is a deliberate experimental design choice rather than a logical circularity. The paper's self-citations (e.g., Mah et al., Tan et al., 2024) provide datasets or prior annotation collections, but these are not used as the justification for the validity of IDEAlign or for the superiority of the triplet format; the empirical comparison is self-contained against external human labels. The bootstrap human-eval oracle is a reliability analysis, not a fitted prediction. There is no self-definitional step, no fitted-input-called-prediction, no load-bearing self-citation, and no imported uniqueness/ansatz. The paper therefore does not reduce to its own inputs, and the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or conceptual entities. Its free parameters are all evaluation-design choices (thresholds, sampling counts, topic and PCA hyperparameters). The key assumptions are about the validity of the triplet-aggregation paradigm and about pooling rankings across source documents. No parameter is fitted to a target quantity in a way that mimics a derived constant.

free parameters (4)
  • Minimum denominator threshold for Eq. (1) = not specified
    The paper states 'We ensure that the denominator in Equation (1) exceeds a certain threshold for all pairs' but does not give the threshold or how it was chosen, affecting which pairs enter the ranking.
  • LLM sampling count for triplet judgments = 10 (for binary/continuous), not specified for triplet
    Section 4.4 says for each pairwise assessment they sample 10 responses and average, but the triplet protocol's query count is not stated.
  • Topic model hyperparameters = LDA topics {50,75,100,125,150}; HDBSCAN min cluster size {2,...,15}
    Appendix B lists these as tunable, and Table 9 searches them to maximize correlation with human labels.
  • PCA post-processing dimension = d = original_dim/100 for Mu et al.; top 1/3 and 2/3 principal components for Raunak et al.
    Appendix A.2 describes these variants, each producing a different similarity metric.
axioms (4)
  • domain assumption Relative odd-one-out judgments are more reliable and meaningful than absolute similarity ratings.
    Section 4.1 motivates the task with citations to comparative-judgment literature (Laming 2003, Pollitt 2012, etc.), but the paper does not independently verify this for the specific educational contexts.
  • domain assumption Aggregating triplet judgments via Eq. (1) yields a valid pairwise similarity measure.
    Equation (1) defines sim(A,B) as the proportion of triplets where neither A nor B is selected as odd; the paper assumes this captures relative similarity without validation against an independent yardstick.
  • domain assumption The same triplet protocol is a valid task for LLMs as for humans.
    Section 4.4 applies the identical odd-one-out prompt to LLMs and treats their responses as comparable to human judgments, an assumption that could fail if LLMs have different decision biases.
  • domain assumption Spearman rank correlation on pooled pair rankings across sources is an appropriate alignment metric.
    Section 4.5 computes one Spearman correlation across all pairs from all transcripts/essays, assuming cross-source commensurability of similarity scores, which is not established.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of IDEAlign: Comparing Large Language Models to Human Experts in Open-ended Interpretive Annotations." pith.science (2026). https://pith.science/paper/7MSW4WLA

@misc{pith2026250902855,
  author       = {Pith},
  title        = {Pith review of: IDEAlign: Comparing Large Language Models to Human Experts in Open-ended Interpretive Annotations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MSW4WLA}},
  note         = {Machine review of arXiv:2509.02855}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) are increasingly applied to open-ended, interpretive annotation tasks, such as thematic analysis by researchers or generating feedback on student work by teachers. These tasks involve free-text annotations requiring expert-level judgments grounded in specific objectives (e.g., research questions or instructional goals). Evaluating whether LLM-generated annotations align with those generated by expert humans is challenging to do at scale, and currently, no validated, scalable measure of similarity in ideas exists. In this paper, we (i) introduce the scalable evaluation of interpretive annotation by LLMs as a critical and understudied task, (ii) propose IDEAlgin, an intuitive benchmarking paradigm for capturing expert similarity ratings via a "pick-the-odd-one-out" triplet judgment task, and (iii) evaluate various similarity metrics, including vector-based ones (topic models, embeddings) and LLM-as-a-judge via IDEAlgin, against these human benchmarks. Applying this approach to two real-world educational datasets (interpretive analysis and feedback generation), we find that vector-based metrics largely fail to capture the nuanced dimensions of similarity meaningful to experts. Prompting LLMs via IDEAlgin significantly improves alignment with expert judgments (9-30% increase) compared to traditional lexical and vector-based metrics. These results establish IDEAlgin as a promising paradigm for evaluating LLMs against open-ended expert annotations at scale, informing responsible deployment of LLMs in education and beyond.

Figures

Figures reproduced from arXiv: 2509.02855 by Dorottya Demszky, Hyunji Nam, James Malamut, Lucia Langlois, Mei Tan.

Figure 1
Figure 1. Figure 1: We introduce IDEAlign, a paradigm for measuring [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: IDEAlign is a benchmarking paradigm for eliciting expert similarity judgments via an intuitive pick-the-odd-one￾out task. Experts are presented with three annotations and asked to select the most dissimilar one according to a specified, domain-relevant criterion. Many such triplet decisions are aggregated to generate pairwise similarity scores for every annotation pair, which can then be compared with auto… view at source ↗
Figure 3
Figure 3. Figure 3: Task interface for IDEAlign benchmarking. Experts [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: We perform 100 bootstrap runs on the human evalu [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task interface for assessing and interpreting stu￾dent’s mathematical understanding from classroom tran￾scripts. We collected annotations from expert math educators using a web-based interface. Participants were presented with a classroom transcript, metadata about the lesson’s purpose, learning goals, and math activities. Their task was to identify specific lines of student talk that signaled meaningful m… view at source ↗
Figure 7
Figure 7. Figure 7: Task interface for collecting expert similarity judgments for the math data. We collected similarity judg￾ments from expert math educators using the pick-the-odd-one￾out via a web-based interface. Participants were presented with a randomized triplet of annotations, generated either by LLMs or human experts, and metadata about the lesson’s purpose and learning goals. Their task was to identify the most dif… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 25 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I

    Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. 2023. https://doi.org/10.1126/science.adi6000 Prediction-powered inference . Science, 382(6671):669--674

    doi:10.1126/science.adi6000 2023

  4. [4]

    Anthropic. 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family: Opus, sonnet, haiku

    work page 2024

  5. [5]

    Anthropic. 2025. https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf System card: Claude opus 4 & claude sonnet 4

    work page 2025

  6. [6]

    David Blei, Andrew Ng, and Michael Jordan. 2001. https://proceedings.neurips.cc/paper_files/paper/2001/file/296472c9542ad4d4788d543508116cbc-Paper.pdf Latent dirichlet allocation . In Advances in Neural Information Processing Systems, volume 14. MIT Press

    work page 2001

  7. [7]

    Ljubi s a Boji \'c , Olga Zagovora, Asta Zelenkauskaite, Vuk Vukovi \'c , Milan C abarkapa, Selma Veseljevi \'c Jerkovi \'c , and Ana Jovan c evi \'c . 2025. Comparing large language models and human annotators in latent content analysis of sentiment, political leaning, emotional intensity and sarcasm. Scientific reports, 15(1):11477

    work page 2025

  8. [8]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324--345

    1952

  9. [9]

    Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-graber, and David Blei. 2009. https://proceedings.neurips.cc/paper_files/paper/2009/file/f92586a25bb3145facd64ab20fd554ff-Paper.pdf Reading tea leaves: How humans interpret topic models . In Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc

    work page 2009

  10. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  11. [11]

    Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://arxiv.org/abs/2007.12626 Summeval: Re-evaluating summarization evaluation . Preprint, arXiv:2007.12626

    Pith/arXiv arXiv 2021

  12. [12]

    Manaal Faruqui and Chris Dyer. 2014. https://doi.org/10.3115/v1/P14-5004 Community evaluation and exchange of word vectors at wordvectors.org . In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 19--24, Baltimore, Maryland. Association for Computational Linguistics

    work page doi:10.3115/v1/p14-5004 2014

  13. [13]

    Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, and John P Dickerson. 2025. https://openreview.net/forum?id=MzHNftnAM1 Style outweighs substance: Failure modes of LLM judges in alignment benchmarking . In The Thirteenth International Conference on Learning Representations

    work page 2025

  14. [14]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821

    Pith/arXiv arXiv 2021

  15. [15]

    Arthur C Graesser, Shulan Lu, George Tanner Jackson, Heather Hite Mitchell, Mathew Ventura, Andrew Olney, and Max M Louwerse. 2004. Autotutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36(2):180--192

    work page 2004

  16. [16]

    Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan. 2022. New and improved embedding model. https://openai.com/index/new-and-improved-embedding-model/

    work page 2022

  17. [17]

    Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794

    Pith/arXiv arXiv 2022

  18. [18]

    Ben Hamner, Jaison Morgan, lynnvandev, Mark Shermis, and Tom Vander Ark. 2012. The hewlett foundation: Automated essay scoring. https://kaggle.com/competitions/asap-aes. Kaggle

    work page 2012

  19. [19]

    Mete Ismayilzada, Claire Stevenson, and Lonneke van der Plas. 2025. https://arxiv.org/abs/2411.02316 Evaluating creative short story generation in humans and large language models . Preprint, arXiv:2411.02316

    Pith/arXiv arXiv 2025

  20. [20]

    Ian Jones and Lara Alcock. 2012. Summative peer assessment of undergraduate calculus using adaptive comparative judgement. Mapping university mathematics assessment practices, pages 63--74

    work page 2012

  21. [21]

    Sal Khan. 2023. https://blog.khanacademy.org/harnessing-ai-so-that-all-students-benefit-a-nonprofit-approach-for-equal-access/ Harnessing gpt-4 so that all students benefit. a nonprofit approach for equal access . Khan Academy Blog

    work page 2023

  22. [22]

    Donald Laming. 2003. Human judgment: The eye of the beholder. Cengage Learning EMEA

    work page 2003

  23. [23]

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. https://arxiv.org/abs/2308.03281 Towards general text embeddings with multi-stage contrastive learning . Preprint, arXiv:2308.03281

    Pith/arXiv arXiv 2023

  24. [24]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  25. [25]

    Chin-Yew Lin and Franz Josef Och. 2004. https://www.aclweb.org/anthology/C04-1072 ORANGE : a method for evaluating automatic evaluation metrics for machine translation . In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics , pages 501--507, Geneva, Switzerland. COLING

    work page 2004

  26. [26]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958

    Pith/arXiv arXiv 2021

  27. [27]

    From sentence-corrections to deeper dialogue: Qualitative insights from llm and teacher feedback on student writing

    Christopher Mah, Mei Tan, Lena Phalen, Alexa Sparks, and Dora Demszky. From sentence-corrections to deeper dialogue: Qualitative insights from llm and teacher feedback on student writing. Available at SSRN 5213040

    work page

  28. [28]

    Gonzalo Martínez, José Alberto Hernández, Javier Conde, Pedro Reviriego, and Elena Merino-Gómez. 2024. https://doi.org/10.1145/3696459 Beware of words: Evaluating the lexical diversity of conversational llms using chatgpt as case study . ACM Transactions on Intelligent Systems and Technology

    work page doi:10.1145/3696459 2024

  29. [29]

    Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit. Http://mallet.cs.umass.edu

    work page 2002

  30. [30]

    Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2018. https://arxiv.org/abs/1702.01417 All-but-the-top: Simple and effective postprocessing for word representations . Preprint, arXiv:1702.01417

    Pith/arXiv arXiv 2018

  31. [31]

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. https://doi.org/10.18653/v1/2023.eacl-main.148 MTEB : Massive text embedding benchmark . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014--2037, Dubrovnik, Croatia. Association for Computational Linguistics

    doi:10.18653/v1/2023.eacl-main.148 2023

  32. [32]

    Alberto Muñoz-Ortiz, Carlos Gómez-Rodríguez, and David Vilares. 2024. https://doi.org/10.1007/s10462-024-10903-2 Contrasting linguistic patterns in human and llm-generated news text . Artificial Intelligence Review, 57(10)

    doi:10.1007/s10462-024-10903-2 2024

  33. [33]

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, P...

    Pith/arXiv arXiv 2022

  34. [34]

    Jun-Ping Ng and Viktoria Abrecht. 2015. https://doi.org/10.18653/v1/D15-1222 Better summarization evaluation with word embeddings for ROUGE . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925--1930, Lisbon, Portugal. Association for Computational Linguistics

    doi:10.18653/v1/d15-1222 2015

  35. [35]

    Zhao, Yi Luan, Keith B

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021. https://arxiv.org/abs/2112.07899 Large dual encoders are generalizable retrievers . Preprint, arXiv:2112.07899

    Pith/arXiv arXiv 2021

  36. [36]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner...

    Pith/arXiv arXiv 2024

  37. [37]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002 a . Bleu: a method for automatic evaluation of machine translation. pages 311--318

    work page 2002

  38. [38]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 b . https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics

    arXiv 2002

  39. [39]

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. https://arxiv.org/abs/2102.01454 Mauve: Measuring the gap between neural text and human text using divergence frontiers . Preprint, arXiv:2102.01454

    Pith/arXiv arXiv 2021

  40. [40]

    Alastair Pollitt. 2012. The method of adaptive comparative judgement. Assessment in Education: principles, policy & practice, 19(3):281--300

    work page 2012

  41. [41]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners

    2019

  42. [42]

    Vikas Raunak, Vaibhav Kumar, Vivek Gupta, and Florian Metze. 2020. https://doi.org/10.18653/v1/2020.repl4nlp-1.19 On dimensional linguistic properties of the word embedding space . In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 156--165, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2020.repl4nlp-1.19 2020

  43. [43]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . Preprint, arXiv:1908.10084

    Pith/arXiv arXiv 2019

  44. [44]

    Jennifer Routh, Sharmini Julita Paramasivam, Peter Cockcroft, Sarah Wood, John Remnant, Corn \'e lie Westermann, Alison Reid, Patricia Pawson, Sheena Warman, Vishna Devi Nadarajah, et al. 2023. Rating and ranking preparedness characteristics important for veterinary workplace clinical training: a novel application of pairwise comparisons and the elo algor...

    work page 2023

  45. [45]

    Kayla Schroeder and Zach Wood-Doughty. 2025. https://arxiv.org/abs/2412.12509 Can you trust llm judgments? reliability of llm-as-a-judge . Preprint, arXiv:2412.12509

    Pith/arXiv arXiv 2025

  46. [46]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857--16867

    2020

  47. [47]

    Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1216 Do LLM s plan like human writers? comparing journalist coverage of press releases with LLM s . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21814--21828, Miami, Florida, USA. Assoc...

    work page doi:10.18653/v1/2024.emnlp-main.1216 2024

  48. [48]

    James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. 2024. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7):1285--1295

    work page 2024

  49. [49]

    Mei Tan, Christopher Mah, and Dorottya Demszky. 2024. Reframing authority: A computational measure of power-affirming feedback on student writing. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pages 417--421

    work page 2024

  50. [50]

    Pina Tarricone and C Paul Newhouse. 2016. Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability. International Journal of Educational Technology in Higher Education, 13(1):16

    work page 2016

  51. [51]

    LearnLM Team, Abhinit Modi, Aditya Srikanth Veerubhotla, Aliya Rysbek, Andrea Huber, Brett Wiltshire, Brian Veprek, Daniel Gillick, Daniel Kasenberg, Derek Ahmed, et al. 2024. Learnlm: Improving gemini for learning. arXiv preprint arXiv:2412.16429

    Pith/arXiv arXiv 2024

  52. [52]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024 a . https://arxiv.org/abs/2212.03533 Text embeddings by weakly-supervised contrastive pre-training . Preprint, arXiv:2212.03533

    Pith/arXiv arXiv 2024

  53. [53]

    Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky

    Rose E. Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. 2024 b . https://arxiv.org/abs/2310.10648 Bridging the novice-expert gap via models of decision-making: A case study on remediating math mistakes . Preprint, arXiv:2310.10648

    Pith/arXiv arXiv 2024

  54. [54]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. https://arxiv.org/abs/2002.10957 Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers . Preprint, arXiv:2002.10957

    Pith/arXiv arXiv 2020

  55. [55]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. https://arxiv.org/abs/2412.13663 Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and lo...

    Pith/arXiv arXiv 2024

  56. [56]

    Hui Wei, Shenghua He, Tian Xia, Fei Liu, Andy Wong, Jingyang Lin, and Mei Han. 2025. https://arxiv.org/abs/2408.13006 Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates . Preprint, arXiv:2408.13006

    Pith/arXiv arXiv 2025

  57. [57]

    Qingsong Wen, Jing Liang, Carles Sierra, Rose Luckin, Richard Tong, Zitao Liu, Peng Cui, and Jiliang Tang. 2024. https://doi.org/10.1145/3637528.3671498 Ai for education (ai4edu): Advancing personalized education with llm and adaptive learning . New York, NY, USA. Association for Computing Machinery

    arXiv 2024

  58. [58]

    Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. https://doi.org/10.18653/v1/2022.naacl-main.47 Reframing human- AI collaboration for generating free-text explanations . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages...

    doi:10.18653/v1/2022.naacl-main.47 2022

  59. [59]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    Pith/arXiv arXiv 2020

  60. [60]

    Meyer, and Steffen Eger

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. https://doi.org/10.18653/v1/D19-1053 M over S core: Text generation evaluating with contextualized embeddings and earth mover distance . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferenc...

    doi:10.18653/v1/d19-1053 2019

  61. [61]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . Preprint, arXiv:2306.05685

    Pith/arXiv arXiv 2023

  62. [62]

    Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. https://aclanthology.org/2024.lrec-main.816/ Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340--9351, Torino, Ital...

    2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.