Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automated nugget scoring of two search-augmented LLM answers separates human win, tie, and loss verdicts: the three score-difference distributions differ at $p \leq 1.2 \times 10^{-24}$.

desk verdict A useful, honest extension of nugget-based evaluation to Search Arena, with a real but addressable circularity concern in nugget generation. read the letter →

arxiv 2504.20006 v2 pith:IQGQQYJN submitted 2025-04-28 cs.IR

Sahel Sharifymoghaddam , Shivani Upadhyay , Nandan Thakur , Ronak Pradeep , Jimmy Lin This is my paper · ORCID
classification cs.IR
keywords nugget-basedevaluationhumanpreferenceLLMretrieval-augmentedgenerationsearcharenaautomaticfactrecallKolmogorov-Smirnovtestdiagnostic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Battles, or side-by-side comparisons where humans pick the better LLM answer, are widely used but tell you only who won, not why. This paper tests whether automatic fact-recall scoring can explain and diagnose those judgments. On roughly 5,100 single-turn battles from the public Search Arena data, it splits each answer into atomic facts, asks a language model which facts a good answer should contain, and scores each answer by how many of those facts it supports. The paper finds that the difference in these scores lines up with human verdicts: the score-difference distributions for A-wins, ties, and B-wins are distinct at $p$-values below $1.2 \times 10^{-24}$, and a simple threshold agrees with the human winner in about 52--55% of decisive battles. The payoff is an evaluation signal that says what fact coverage drove the win and which query types (ambiguous, assumptive, multi-faceted) resist fact-based scoring.

What carries the argument

The central machinery is the automatic nuggetization pipeline, which (1) uses an LLM to generate atomic facts, or nuggets, from the query, the top retrieved chunks, and the two candidate responses, labeling each nugget vital or okay; (2) assigns each response a support label of support, partial support, or no support for every nugget; and (3) aggregates an All Score that counts nuggets of all importance and support levels. The paper then feeds the per-battle score differences into kernel density estimates, pairwise Kolmogorov-Smirnov tests, and a confusion matrix with a 0.07 tie threshold to relate the score to human verdicts. That pipeline is what converts raw preference judgments into a structured, per-fact explanation of why one answer won.

What would settle it

Take a random sample of, say, 300 single-turn battles, have two human annotators independently list the atomic facts a good answer should contain and score the same two model responses, then compare those human-nugget score differences across the three human-vote groups; if the separation disappears or if the automatic support labels disagree with human labels on more than a small fraction of nuggets, the central correlation is an artifact of the automatic nuggetizer rather than a property of answer content.

Watch

Extended reading notes

Core claim

Conditioned on the human verdict, the distribution of nugget score difference $score_B - score_A$ is not the same across the three outcomes: it shifts left when model A wins, centers near zero on ties, and shifts right when model B wins. Pairwise Kolmogorov-Smirnov tests give statistics 0.205 to 0.313 with $p$-values at most $1.2 \times 10^{-24}$, so the paper concludes that nugget score differences align with human preferences in head-to-head evaluation of search-augmented LLMs. With a tie threshold of 0.07 on the score difference, the automatic preference matches the human winner in 54.7% of model-A-winning battles and 52.5% of model-B-winning battles; disagreements, called preference inversions, concentrate in ambiguous, assumptive, and multi-faceted queries, with German-language battles showing the highest inversion rate. The paper also reports that generating nuggets from the two LLM responses alone, without the scraped URL contents, gives essentially the same agreement with human preferences as the full pipeline.

Load-bearing premise

The argument assumes the automatically extracted nuggets are a faithful, unbiased inventory of the facts a good answer must contain, without manual verification on this dataset, so if those labels are wrong or biased toward the scoring model's own style, the observed alignment with human preferences could be an artifact.

Editorial extensions

If this is right

  • Automated nugget scoring can serve as an explanatory overlay on arena battle data, showing which answer contained more of the facts a good answer needs.
  • The method is diagnostic: inversion rates identify query types and languages where human and nugget preferences diverge, pointing to where evaluation of RAG systems needs refinement.
  • Because nuggetization from LLM responses alone matches the version that uses scraped URL contents, the approach transfers to settings without reliable external evidence.
  • Compared with an LLM-as-a-judge baseline, nugget scoring produces fewer preference inversions and does not collapse the tie category, so it offers a structured complement to direct judging.
  • The score-difference distributions provide a principled basis for setting tie thresholds, since the threshold can be tuned to match the observed human tie rate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported roughly 50--55% agreement understates the signal in the score; with calibrated thresholds or a proper scoring rule, the nugget score difference could be converted into a probabilistic predictor of human choice and compared against other predictors explicitly.
  • Because the 0.07 tie threshold and the distribution separation are fit on this dataset, applying the same nuggetization to a new arena will likely require its own threshold calibration, otherwise the apparent alignment may shrink.
  • The inversion pattern suggests a possible two-stage evaluation: use nugget scores for knowledge-intensive queries, which show the highest alignment, and fall back to human judgment or a different metric for ambiguous and multi-faceted queries.
  • The finding that responses-only nuggetization preserves agreement implies the method could be applied retroactively to standard arena battles that lack retrieved documents, turning historical preference data into a diagnostic signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper applies the AutoNuggetizer framework to 5,103 single-turn battles from LMArena's Search Arena v1 dataset. For each battle, it generates information nuggets from the query, scraped URL contents, and the two model responses; assigns support labels to each response; computes a nugget score; and compares the score difference with the human win/tie judgment. The authors report that the three conditional distributions of nugget score differences are significantly different (pairwise KS p-values ≤ 1.2e−24), that a confusion-matrix comparison with a tie threshold of 0.07 yields 54.7%/52.5% agreement on model-A/model-B wins, that inversion rates vary across query classes and languages, and that using only model responses for nugget generation gives comparable agreement. They also benchmark an LLM-as-a-judge baseline, which agrees more on clear wins but rarely predicts ties. The paper argues that these results demonstrate the explanatory and diagnostic value of nugget-based evaluation.

Significance. If the headline results are taken at face value, the paper offers a scalable, fully automatic way to turn preference judgments into component-level explanations: nugget recall differences partially track human win/tie decisions, and the per-nugget support labels could diagnose retrieval versus generation failures. The release of code and the use of public LMArena data are strengths, as is the explicit comparison to an LLM-as-a-judge baseline. However, the central quantitative claims—the agreement rates and 'strong alignment'—rest on an in-sample tuned tie threshold, a post hoc metric choice, and a nugget-generation step that includes the very responses being scored. These issues must be resolved before the explanatory-power claim is fully supported.

major comments (4)
  1. [Section 4, Figure 4] The tie threshold of 0.07 is selected by sweeping values between 0.05 and 0.15 on the same human labels used to compute the agreement rates: 'This threshold was selected by sweeping values between 0.05 and 0.15 in increments of 0.01... to most closely reflect an equal distribution of modelA wins, modelB wins, and ties when the human preference is a tie.' This is in-sample tuning; the reported diagonal percentages (54.7% and 52.5%) are therefore optimistic estimates, and no confidence intervals are provided. Please report the agreement rates across the full threshold sweep, use held-out data or a pre-specified threshold, and include bootstrap or exact binomial confidence intervals for the reported agreement rates.
  2. [Section 3, Nugget generation] The AutoNuggetizer prompt includes 'the responses from each model' in addition to the query and retrieved chunks. Because the nugget set is generated from the two candidate answers, the subsequent assignment step credits each model for content that the generator selected partly because it appeared in that model's response. This makes the score difference d = score_B − score_A a partially self-referential quantity, not a pure measure of recall of facts a good answer must contain. Section 4.3 ablates URL contents but never removes the candidate responses from the generation prompt, so the direction and size of this bias are unknown. Please run an ablation that generates nuggets from the query and retrieved chunks only (with a fallback for battles with unusable URLs) and compare the KS results and agreement rates.
  3. [Section 3, Nugget evaluation] The choice of 'All Score' over 'Strict Vital' and the other two combinations is justified post hoc on the same data: 'we adopt the "All Score" metric, which achieves the highest recall... the "Strict Vital" metric... is too strict for our use case.' This metric selection, like the tie-threshold tuning, uses the human votes that are later used to measure agreement. Please report results for all four metric combinations, or justify the metric a priori, so that the reported alignment is not a product of selection.
  4. [Sections 5 and 7] The text describes the results as 'strong alignment' and 'strongly correlated' (Section 5) and 'demonstrate a strong alignment' (Section 7), yet the headline agreement rates are 54.7% and 52.5% for clear winners, with lower rates for ties, and no correlation coefficient is computed. Please qualify these claims and include a comparison to a trivial baseline (e.g., always predicting the majority class) to calibrate what these agreement rates mean.
minor comments (5)
  1. [Figure 1] Figure 1 contains typos: 'Answer Reponse' and 'longetivity' in the displayed response text.
  2. [Figure 2] The KDE bandwidth of 0.5 is not justified, and the axis labels repeat 'Density' without a clear quantity; a brief note on bandwidth selection would improve reproducibility.
  3. [Section 4, Figure 2 vs Figure 4] Battles where both responses are 'Tie (both bad)' are excluded from the distributions in Figure 2 but are included in the confusion matrices in Figures 4 and 8; please clarify the treatment of this category consistently across analyses.
  4. [Section 4.2] The statement 'assuming a similar distribution of query categories across languages' is made without supporting evidence; since language-specific query classification is deferred, this assumption should be flagged more explicitly as an untested assumption.
  5. [Reference [25]] The venue description for Rackauckas et al. appears garbled: 'co-located with 10th International Conference on Online Publishing (SIGIR 2024)' is not a recognizable conference name; please correct the reference.

Circularity Check

2 steps flagged · score 6.0 of 10

Nugget-score 'predictions' are partly in-sample: the tie threshold is fitted to the same human votes used for the confusion matrix, and the nugget set is generated from the very responses being scored.

  1. self definitional [Section 3, 'Nugget evaluation', nugget generation step; also Figure 1 caption.]
    "we construct a request to AutoNuggetizer that includes the query (i.e., the prompt itself), along with relevant chunks retrieved from our created corpus, ordered by relevance. In addition to these chunks, we include the responses from each model, inserted in random order to mitigate the positional bias."

    The nugget set is generated from the same two model responses that are subsequently scored against it. If a fact appears only in model A's answer, it can be emitted as a nugget and then credited to model A during assignment, so score_A partially measures self-consistency with the nuggetizer's extraction rather than recall of an externally defined gold set. The difference d = score_B - score_A therefore depends on which response contributed content to the nugget set, which can inflate the distributional separation in Figures 2-3 and the diagonal agreement in Figure 4. The Section 4.3 ablation removes URL contents but never removes the candidate responses from generation, so the direction and size of the dependence are untested.

  2. fitted input called prediction [Section 4, Figure 4 and surrounding text; Section 5 prediction claim.]
    "For 'nugget preference', we use a threshold of 0.07, meaning that when the nugget score difference between the two model outputs falls within ± 0.07, the comparison is considered a tie. This threshold was selected by sweeping values between 0.05 and 0.15 in increments of 0.01. A threshold of 0.07 results in nugget preferences that most closely reflect an equal distribution of modelA wins, modelB wins, and ties when the human preference is a tie (second row in Figure 4)."

    The tie threshold is chosen by optimizing a criterion on the same human preference labels that are later used to build the confusion matrix and to report agreement and inversion rates. Because no held-out split or nested selection is described, the diagonal percentages (e.g., 938/1714 and 924/1761) and the 'predict human preferences over 50% of the time' claim are in-sample evaluations of a threshold fitted to those labels. The threshold-free KS tests in Figures 2-3 remain independent, so the circularity is partial rather than total.

full rationale

The two flagged steps are the load-bearing sources of circularity. First, the nugget generation prompt includes the candidate responses, so the 'gold' nugget set is not independent of the answers being scored; a fact unique to one response can become a nugget and then be credited to that same response, making the score difference partially self-referential. Second, the 0.07 tie threshold used to convert continuous nugget score differences into predicted win/tie labels is selected by sweeping against the human tie row of the very confusion matrix that is then reported as predictive evidence. This is a fitted-input-called-prediction pattern: the same human labels select the threshold and evaluate the resulting predictions. The Kolmogorov-Smirnov distributional results in Figures 2 and 3 are threshold-free and therefore retain independent content, and the correlation with human preferences is not purely definitional, which is why the score is 6 rather than 8 or 10. The self-citations to the AutoNuggetizer framework are not themselves circular because the framework is an implemented tool with prior TREC validation; the circularity lies in the present pipeline's test-label-dependent threshold and response-dependent nugget generation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on the authors' prior AutoNuggetizer with no free parameters in the nugget scoring itself, but the evaluation pipeline adds three hand-picked thresholds and a post hoc metric choice. No new physical or conceptual entities are introduced.

free parameters (3)
  • tie_threshold_07 = 0.07
    Swept between 0.05 and 0.15 in increments of 0.01 to best match the human tie distribution on the same 5,103 battles; used to binarize nugget score differences in Figure 4.
  • tie_threshold_01 = 0.1
    Used for the no-URL variant in Figure 7; chosen because scores are more discrete without URL contents, not independently tuned.
  • query_class_threshold_7 = 7
    Queries with a maximum GPT-4.1 attribute score below 7 are excluded from query classification; this cut is arbitrary and affects the reported inversion rates per query class.
assumptions (4)
  • domain assumption Human votes in Search Arena are valid ground truth for response quality.
    The evaluation defines inversion as disagreement with human votes and treats human preference as the target of prediction.
  • domain assumption Scraped URL content approximates what the RAG systems retrieved when answering.
    Section 3 corpus generation downloads URLs at evaluation time; the Limitations section acknowledges web content is dynamic and scrapes can fail.
  • domain assumption GPT-4.1 can generate and assign nuggets reliably across languages.
    Section 4.2 notes AutoNuggetizer 'had only been run on English responses', yet it is applied here to German, Chinese, Russian, etc., without external validation.
  • ad hoc to paper The 'All Score' metric, counting all nuggets with partial support, is appropriate.
    Section 3 states they adopt All Score because it 'achieves the highest recall', an in-sample metric choice rather than a pre-specified one.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses." pith.science (2026). https://pith.science/paper/IQGQQYJN

@misc{pith2026250420006,
  author       = {Pith},
  title        = {Pith review of: Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IQGQQYJN}},
  note         = {Machine review of arXiv:2504.20006}
}
read the original abstract

Battles, or side-by-side comparisons in so-called arenas that elicit human preferences, have emerged as a popular approach for assessing the output quality of LLMs. Recently, this idea has been extended to retrieval-augmented generation (RAG) systems. While undoubtedly representing an advance in evaluation, battles have at least two drawbacks, particularly in the context of complex information-seeking queries: they are neither explanatory nor diagnostic. Recently, the nugget evaluation methodology has emerged as a promising approach to evaluate the quality of RAG answers. Nuggets decompose long-form LLM-generated answers into atomic facts, highlighting important pieces of information necessary in a "good" response. In this work, we apply our AutoNuggetizer framework to analyze data from roughly 7K Search Arena battles provided by LMArena in a fully automatic manner. Our results show a significant correlation between nugget scores and human preferences, showcasing promise in our approach to explainable and diagnostic system evaluations. All the code necessary to reproduce results in our work is available in https://github.com/castorini/lmsys_nuggetize.

Figures

Figures reproduced from arXiv: 2504.20006 by the authors.

Figure 1
Figure 1. An end-to-end example from Search Arena illustrating both nugget generation and assign [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Empirical probability density function (PDF) of nugget score differences ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Empirical cumulative distribution functions (CDFs) comparing nugget score differences [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Confusion matrix comparing human and nugget preferences. A thresh￾old of 0.07 is applied to treat nugget pref￾erence scores as a tie [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Confusion matrices comparing human and nugget preferences across eight query classes [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Confusion matrices comparing human and nugget preferences across six different languages [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Confusion matrix comparing human and nugget preferences, using LLM responses as the sole source for nugget generation. A threshold of 0.1 is applied to treat nugget preference scores as a tie. To assess the impact of scraped URL contents on the effec￾tiveness of nugget…
Figure 8
Figure 8. Figure 8: Confusion matrix comparing human and GPT4.1 preferences [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Dataset Overview: (left) winners distribution; (right) language distribution. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Histogram showing the classified attributes for 5,103 single-turn queries in the Search [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Prompt used by GPT4.1 judge to evaluate the model answers in Search Arena. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.

Reference graph

Works this paper leans on

32 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Conversational gold: Evaluating personalized conversational search system using gold nuggets

    Zahra Abbasiantaeb, Simon Lupart, Leif Azzopardi, Jeffery Dalton, and Mohammad Alianne- jadi. Conversational gold: Evaluating personalized conversational search system using gold nuggets. preprint arXiv:2503.09902, 2025. 2 9

    arXiv 2025

  2. [2]

    Generative information retrieval evaluation

    Marwah Alaofi, Negar Arabzadeh, Charles LA Clarke, and Mark Sanderson. Generative information retrieval evaluation. In Information Access in the Era of Generative AI , pages 135–159. 2024. 2

    work page 2024

  3. [3]

    A comparison of methods for evaluating generative IR

    Negar Arabzadeh and Charles LA Clarke. A comparison of methods for evaluating generative IR. preprint arXiv:2404.04044, 2024. 2

    arXiv 2024

  4. [4]

    Chatbot arena: An open platform for evaluating LLMs by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference on Machine Learning, 2024. 1, 3

    work page 2024

  5. [5]

    Different structures for evaluating answers to complex questions: pyramids won’t topple, and neither will human assessors

    Hoa Trang Dang and Jimmy Lin. Different structures for evaluating answers to complex questions: pyramids won’t topple, and neither will human assessors. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL 2007), pages 768–775, Prague, Czech Republic, 2007. 2

    work page 2007

  6. [6]

    Pencils down! automatic rubric-based evaluation of re- trieve/generate systems

    Naghmeh Farzi and Laura Dietz. Pencils down! automatic rubric-based evaluation of re- trieve/generate systems. In Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR ’24, pages 175–184, Washington, D.C., 2024. 2

    work page 2024

  7. [7]

    RAG-QA arena: Evaluating domain robustness for long-form retrieval augmented question answering

    Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min, and Vittorio Castelli. RAG-QA arena: Evaluating domain robustness for long-form retrieval augmented question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4354–4374, Miami, Florida, 2024. 1

    work page 2024

  8. [8]

    GenAI arena: An open evaluation platform for generative models

    Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. GenAI arena: An open evaluation platform for generative models. Advances in Neural Information Processing Systems, 37:79889–79908, 2024. 1

    work page 2024

Show all 32 references
  1. [9]

    Automatically evaluating answers to definition ques- tions

    Jimmy Lin and Dina Demner-Fushman. Automatically evaluating answers to definition ques- tions. In Proceedings of the 2005 Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005), pages 931–938, Vancouver, Canada, 2005. 2

    2005

  2. [10]

    Methods for automatically evaluating answers to complex questions

    Jimmy Lin and Dina Demner-Fushman. Methods for automatically evaluating answers to complex questions. Information Retrieval, 9(5):565–587, 2006. 2

    2006

  3. [11]

    Will pyramids built of nuggets topple over? In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 383–390, New York, New York, 2006

    Jimmy Lin and Dina Demner-Fushman. Will pyramids built of nuggets topple over? In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 383–390, New York, New York, 2006. 2

    2006

  4. [12]

    Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference...

    2021

  5. [13]

    Deconstructing nuggets: the stability and reliability of complex question answering evaluation

    Jimmy Lin and Pengyi Zhang. Deconstructing nuggets: the stability and reliability of complex question answering evaluation. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007), pages 327– 334, A...

    2007

  6. [14]

    Wildvision: Evaluating vision-language models in the wild with human preferences

    Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences. preprint arXiv:2406.11069, 2024. 1

    2024 arXiv

  7. [15]

    Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, and Noah Hibbler

    James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, and Noah Hibbler. On the evaluation of machine-generated reports. In Proceedings of the 47th International ...

    2024

  8. [16]

    MTEB arena

    Hugging Face . MTEB arena. https://huggingface.co/spaces/mteb/arena, 2023. Ac- cessed: 2025-04-24. 2

    2023

  9. [17]

    FActScore: Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Met...

    2023

  10. [18]

    Angelopoulos, Wei-Lin Chiang, Narges Norouzi, and Joseph E

    Mihran Miroyan, Tsung-Han Wu, Logan Kenneth King, Tianle Li, Anastasios N. Angelopoulos, Wei-Lin Chiang, Narges Norouzi, and Joseph E. Gonzalez. Introducing the search arena: Evalu- ating search-enabled AI. https://blog.lmarena.ai/blog/2025/search-arena/, 2025. 1, 2, 3

    2025

  11. [19]

    MTEB: Massive text embedding benchmark

    Niklas Muennighoff, Alexandre Tazi, Tom Magister, Mohammad Shoeybi, Teven Le Scao, Dragomir Radev, and Alex Wang. MTEB: Massive text embedding benchmark. preprint arXiv:2302.08968, 2023. 2

    2023 arXiv

  12. [20]

    Evaluating content selection in summarization: The pyramid method

    Ani Nenkova and Rebecca Passonneau. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Bost...

    2004

  13. [21]

    Golbus, and Javed A

    Virgil Pavlu, Shahzad Rajput, Peter B. Golbus, and Javed A. Aslam. IR system evaluation using nugget-based test collections. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM 2012), pages 393–402, Seattle, Washington, 2012. 2

    2012

  14. [22]

    Ragnarök: A reusable RAG framework and baselines for TREC 2024 retrieval-augmented generation track

    Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. Ragnarök: A reusable RAG framework and baselines for TREC 2024 retrieval-augmented generation track. In Advances in Information Retrieval, pages 132–148,...

    2024

  15. [23]

    Initial nugget evaluation results for the TREC 2024 RAG Track with the AutoNuggetizer Framework

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. Initial nugget evaluation results for the TREC 2024 RAG Track with the AutoNuggetizer Framework. preprint arXiv:2411.09607, 2024. 2, 3

    2024 arXiv

  16. [24]

    The great nugget recall: Automating fact extraction and RAG evaluation with large language models

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. The great nugget recall: Automating fact extraction and RAG evaluation with large language models. preprint arXiv:2504.15068, 2025. 2, 4

    2025 arXiv

  17. [25]

    Evaluating rag-fusion with ragelo: an automated elo-based framework

    Zackary Rackauckas, Arthur Câmara, and Jakub Zavrel. Evaluating rag-fusion with ragelo: an automated elo-based framework. In Clemencia Siro, Mohammad Aliannejadi, Hossein A. Rah- mani, Nick Craswell, Charles L. A. Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, and Emi...

    2024

  18. [26]

    Researchy questions: A dataset of multi-perspective, decompositional questions for LLM web agents

    Corby Rosset, Ho-Lam Chung, Guanghui Qin, Ethan C Chau, Zhuo Feng, Ahmed Awadal- lah, Jennifer Neville, and Nikhil Rao. Researchy questions: A dataset of multi-perspective, decompositional questions for LLM web agents. preprint arXiv:2402.17896, 2024. 6, 13

    2024 arXiv

  19. [27]

    MIRAGE-bench: Automatic multilingual benchmark arena for retrieval-augmented generation systems

    Nandan Thakur, Suleman Kazi, Ge Luo, Jimmy Lin, and Amin Ahmad. MIRAGE-bench: Automatic multilingual benchmark arena for retrieval-augmented generation systems. preprint arXiv:2410.13716, 2025. 1

    2025 arXiv

  20. [28]

    Freshstack: Building realistic benchmarks for evaluating retrieval on technical documents

    Nandan Thakur, Jimmy Lin, Sam Havens, Michael Carbin, Omar Khattab, and Andrew Drozdov. Freshstack: Building realistic benchmarks for evaluating retrieval on technical documents. preprint arXiv:2504.13128, 2025. 2

    2025 arXiv

  21. [29]

    V oorhees

    Ellen M. V oorhees. Evaluating answers to definition questions. InCompanion Volume of the Proceedings of HLT-NAACL 2003 — Short Papers, pages 109–111, Edmonton, Canada, 2003. 2 11

    2003

  22. [30]

    V oorhees

    Ellen M. V oorhees. Overview of the TREC 2003 question answering track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003), pages 54–68, Gaithersburg, Maryland,

    2003

  23. [31]

    Gonzalez, Ion Stoica, and Shishir G

    Nithik Yekollu, Arth Bohra, Ashwin Chirumamilla, Kai Wen, Sai Kolasani Wei-Lin Chiang, Anastasios Angelopoulos, Joseph E. Gonzalez, Ion Stoica, and Shishir G. Patil. Agent arena. https://gorilla.cs.berkeley.edu/blogs/14_agent_arena.html, 2024. 1

    2024

  24. [32]

    [[A]]" if assistant A is better,

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Process...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.