{"id":"446ef172-2648-43f5-b399-8f26a4c4892f","arxiv_id":"2504.15689","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new corpus and two crowdsourcing studies show human pairwise judgments are viable for RAG evaluation, whereas LLM-as-judge and reference-based metrics do not reproduce human preferences.","lead":"Researchers crowdsourced 903 human-written RAG answers and 47,320 pairwise quality judgments from 420 workers, comparing them with GPT-4o outputs. They conclude that human pairwise judgments are a reliable way to evaluate RAG systems, while LLM judging and automated reference-based metrics fail.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold labels are defined after deleting the bottom 25% of workers and ~20% of pairs that lack a majority vote; because the same majority vote defines 'reliable,' the cleaned labels may bias all downstream system rankings.","rationale":"The paper is a transparent, well-documented empirical study and releases its data and code, which is real support. The most load-bearing link in the argument is not the LLM configuration (acknowledged as a limitation), nor the absence of a formal cost-effectiveness model (secondary), but the construction of the gold standard. All comparisons that ground the headline—human reliability, LLM-as-judge failure, reference-metric failure—are evaluated against labels that exist only after two post hoc exclusions. The second exclusion is especially concerning because the criterion for exclusion (no majority vote) is the same criterion used to define the label; using it to remove data and then reporting high agreement does not establish reliability. A re-analysis on the full pool, or at minimum balance checks on the excluded pairs, would settle whether the findings are robust. Hence the conditional verdict: accept the empirical contribution, but require the robustness check before the broad reliability/cost-effectiveness claim is used as guidance for RAG evaluation practice.","tokens_in":19872,"tokens_out":6437,"duration_ms":62176,"concrete_test":"Recompute Tables 7, 8, and 9 using gold labels from MACE-weighted majority votes on all 1,352 pairs (retain the worker-competency correction but do not drop minimally differentiable pairs). Also fit a logistic regression predicting exclusion (worker-removal or minimal-differentiability) from pair-origin type (H-H, H-L, L-L), style pair, and utility dimension. If the LLM-over-human grade gap and the Table 8/9 correlations are materially unchanged and exclusion odds are balanced, the cleaning is neutral; if rankings shift or exclusions are imbalanced, the headline claim must be restricted to the cleaned subset.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim ('Human pairwise judgments provide reliable and cost-effective results...') depends on the gold labels constructed in §5.1. Raw agreement is low (mean Krippendorff α=0.19, Table 4). The paper raises it to 0.48 by (i) removing the bottom quarter of workers by MACE competency and (ii) removing 290 of 1,352 pairs (≈20%) classified as 'minimally differentiable' because they lack a 3-of-5 majority vote on ≥4 of 7 dimensions. These cleaned labels are then used for the Bradley-Terry grades in Table 7, the reference-metric correlations in Table 8, and the LLM-judge agreement in Table 9. The problem is structural: 'minimally differentiable' is defined by the same majority-vote procedure that generates the gold label, so deleting those pairs and then reporting high agreement is circular. The expert check in §3.3.2 was run only on 30 deliberately hard pairs and cannot establish that the 20% exclusion is neutral. If excluded pairs are more likely to involve particular origins (e.g., human-vs-human or LLM-vs-LLM), discourse styles, or utility dimensions, then the LLM>human grade advantage and the low metric/LLM correlations could be artifacts of the remaining 78.5% of the data. The paper reports no balance statistics for the excluded pairs, so the reliability claim has not been shown to generalize to the full judgment pool.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CrowdRAG-25, a corpus of 903 human-written and 903 LLM-generated RAG responses for 301 TREC RAG'24 topics across three discourse styles, together with 47,320 human pairwise judgments and 10,556 LLM pairwise judgments for a subset of 65 topics over seven utility dimensions. It reports analyses of human versus LLM writing behavior, an assessment of crowd judgment reliability using Krippendorff's alpha and MACE-based competency correction, and comparisons of crowd judgments with LLM-as-judge and reference-based evaluation metrics. The headline claim is that human pairwise judgments are reliable and cost-effective for RAG evaluation, whereas LLM-based pairwise judgments and automated reference-based metrics fail to reproduce human preferences. All data and code are released openly.","tokens_in":20331,"tokens_out":3679,"duration_ms":35882,"significance":"The released corpus is a substantial and potentially reusable resource, and the study is unusually transparent in its cost accounting, worker recruitment, spam controls, and interaction logging. The writing-behavior analyses (Section 4) are informative and largely independent of the reliability claim. However, the central reliability claim is currently conditional: it is established only after post hoc exclusions, and the exclusion rule is entangled with the gold-label definition. If the authors add robustness and sensitivity analyses demonstrating that the exclusions do not bias the system rankings, the paper would make a solid contribution to RAG evaluation methodology and to the debate on LLM-as-judge.","major_comments":[{"comment":"The reliability claim rests on a circular exclusion. The 'minimally differentiable' split removes pairs that lack a 3-of-5 majority vote on at least 4 of 7 dimensions, and the same majority-vote procedure then defines the gold label for the remaining pairs. Reporting alpha after this exclusion (0.41 to 0.48) as evidence that the gold labels are reliable does not establish reliability on the full judgment pool, and all downstream comparisons in Tables 7-9 inherit this selection. The paper should report balance statistics for the 290 excluded pairs (by response origin, discourse style, topic, and utility dimension), repeat the Bradley-Terry grades, reference-metric correlations, and LLM-judge agreements without the exclusion or under alternative exclusion rules, and show that the rankings are stable. The 30-item expert validation in Section 3.3.2 is too small and deliberately hard to establish neutrality of the exclusion.","section":"Section 5.1, Table 4"},{"comment":"The paper states that approximately the lower quarter of workers by MACE competency are removed to obtain the competency-corrected labels, but it does not specify the exact competency threshold, the number of workers removed, or the number of judgments discarded. Without these details, the cost-per-gold-judgment figure in Table 1 and the reliability improvement in Table 4 cannot be reproduced. Please report the full filtering pipeline and a sensitivity analysis over the exclusion percentile, since the headline claim that crowdsourcing is 'cost-effective' depends directly on this threshold.","section":"Section 5.1, worker exclusion"},{"comment":"The expert validation is not quantitative enough to support the claim that the exclusions are neutral or that there is no systematic task failure. The text says experts demonstrate higher absolute agreement, yet the expert alpha values in Table 4 for the minimally differentiable split are negative (e.g., -0.11 for topical correctness), which appears inconsistent and needs clarification. Moreover, the experts judged only 30 deliberately hard pairs; what is needed is expert-crowd agreement on a random sample of both excluded and retained pairs, with reported agreement coefficients and confidence intervals, before the cleaned gold labels can be treated as ground truth for Tables 7-9.","section":"Sections 3.3.2 and 5.1, Table 4"}],"minor_comments":[{"comment":"The sentence 'no systematic investigation of the capabilities and limitations of human-sourced ground-truth data for retrieval-augmented generation has not done so far' contains a double negative and should be rewritten.","section":"Section 1, Introduction"},{"comment":"The table rows for the Jaccard coefficient and Spearman's rho are not explicitly labeled beyond the legend symbols below the table; adding row labels such as 'Human vs. LLM', 'Document ranking vs. citation set', and 'Human-cited vs. LLM-cited' would improve readability.","section":"Table 3"},{"comment":"The pointwise column header 'Pntw.' reports nItems = 410, but Section 3.3.1 states that 1,645 pointwise judgments were collected for 47 responses; the relationship between these numbers should be explained.","section":"Table 4"},{"comment":"The column headers in Table 9 are garbled in the rendering, with alpha and rho symbols and subscripts not clearly separated; please reformat the header rows so that each agreement and correlation coefficient is unambiguously labeled.","section":"Table 9"},{"comment":"The Spearman correlations in Table 8 are reported without significance tests or confidence intervals; given the small per-topic sample size (six responses per topic), a bootstrap or permutation interval would help assess whether the differences across metrics are meaningful.","section":"Section 5.3, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the circularity of the minimally differentiable exclusion and the absence of sensitivity analyses. This is fixable within the manuscript's scope: the authors already have the raw judgment data, so they can rerun the main analyses without the exclusion and report balance statistics. I do not see grounds for rejection, but the paper should not be accepted until this is addressed. The paper's fit for a SIGIR/IR venue is strong given the release of the corpus and the relevance to RAG evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper contributes a genuinely new resource: CrowdRAG-25, with 1,806 RAG responses (half human, half GPT-4o) across three discourse styles, plus 47,320 pairwise human judgments and 10,556 LLM judgments on seven utility dimensions for 65 topics. The dataset is public, and the writing-behavior analysis—citation patterns, text reuse, readability—is careful and interesting. Second, the paper also delivers a useful negative result: GPT-4o as a pairwise judge shows very low agreement with human gold labels, and reference-based metrics (BERTScore, BLEU, ROUGE) correlate weakly with human preferences. Those findings deserve attention.\n\nThe soft spot is the reliability claim. The abstract says human pairwise judgments are 'reliable and cost-effective,' but the reliability number (Krippendorff α≈0.48) is reached only after removing the bottom quarter of workers by MACE competency and excluding about 20% of pairs classified as 'minimally differentiable'—defined as lacking a 3-of-5 majority on at least 4 of 7 dimensions. That exclusion uses the same majority-vote machinery that defines the gold label, so the improvement from α=0.19 is partly circular. The expert check on 30 hard pairs doesn't resolve it: the experts also disagreed on those pairs, which shows they are hard, but not that the exclusion is neutral across response origin, style, or dimension. The paper reports no balance statistics for the excluded pairs. Since the cleaned labels feed every downstream comparison (Bradley-Terry grades, metric correlations, LLM agreement), the central claim is more conditional than the abstract implies.\n\nThe paper is transparent about all of this, and the writing study and corpus are solid. The cost-effectiveness claim is more of a reporting than a formal comparison, but the numbers are useful. Citation pattern is fine; [13] is their own framework, but it is the natural basis for the dimensions.\n\nBottom line: this deserves a serious referee, but the reliability claim needs reworking. The cleanest path is to also report results on the full judgment pool, justify the exclusion with an independent measure of pair similarity, and show balance stats for excluded pairs. As is, I'd call it a strong contribution with an overstated headline. Recommendation: send to peer review, expect major revision.","headline":"Valuable crowdsourced RAG evaluation corpus and solid negative findings, but the reliability claim rests on a circular post-hoc exclusion that needs to be unwound.","tokens_in":20702,"tokens_out":4364,"would_cite":true,"duration_ms":40422,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human pairwise judgments, not LLM judges or overlap metrics, are the reliable gold standard for RAG response evaluation.","keywords":["retrieval-augmented generation","crowdsourcing","RAG evaluation","pairwise judgments","LLM-as-judge","reference-based metrics","utility dimensions","human preference judgments"],"falsifier":"Take the full 47,320 pairwise judgments, recompute per-topic rankings with no competency filtering and no exclusion of minimally differentiable pairs, and compare the resulting system and style orderings to the paper's cleaned rankings; if the human-versus-LLM advantage or the bullet-style advantage disappears under a plain majority vote, then those findings depend on the cleaning step rather than on the raw crowd signal. A complementary check is to have expert judges label the excluded near-tie pairs and test whether the cleaned gold labels agree with experts on exactly those items.","tokens_in":19719,"feed_emoji":"👥","tokens_out":8064,"duration_ms":69113,"temperature":0.7,"pith_summary":"This paper asks whether ordinary crowd workers can supply the two things RAG evaluation needs — reference responses to compare against, and judgments of response quality — and whether those crowdsourced products are trustworthy. Its central claim is that human pairwise judgments are the reliable and cost-effective source of ground truth: after filtering out low-competency workers and near-identical response pairs, the remaining five-judge majority votes reach agreement levels on par with earlier information-retrieval studies, at a cost of roughly six cents per gold judgment. The same experiments show that LLM-generated responses are judged better than human-written ones on most utility dimensions, that bullet-style responses are preferred over essay and news styles, and that neither LLM-based pairwise judging nor reference-overlap metrics such as BLEU, ROUGE-L, and BERTScore reproduce the human rankings. The authors caution that the single topic set, the single LLM configuration, and the crowd-worker populations limit how far the results generalize.","feed_headline":"Crowdsourced pairwise judgments beat LLM judges for RAG evaluation","feed_subtitle":"47,320 human pairwise judgments outperform LLM judging and overlap metrics at reproducing quality preferences.","key_machinery":"The load-bearing machinery is the pairwise judgment protocol combined with a two-stage cleaning step. Each comparison asks five independent workers to choose between two responses on a seven-dimension rubric, with a neutral option except for overall quality; this replicates the finding of prior work that pairwise formats yield more reliable text-quality judgments than pointwise scales. A competency-weighted majority-vote model estimates per-worker reliability, and judgments from the bottom quarter of workers are dropped; pairs that lack a majority vote on most dimensions are marked as minimally differentiable and excluded from the gold set. Per-topic response rankings are then derived with a probabilistic pairwise-comparison model, and these rankings serve as the ground truth against which LLM judgments and reference-overlap metrics are tested.","core_discovery":"The paper's discovery is that pairwise comparison — showing a worker two responses and asking which is better on a specific dimension, with a neutral option — is the design that makes crowdsourced RAG evaluation work. Across 65 topics, six responses per topic (three human, three LLM) were paired into 1,352 comparison items and judged by five workers each on seven dimensions: topical correctness, logical and stylistic coherence, broad and deep coverage, internal consistency, and overall quality. Agreement among raw crowd judgments starts low, averaging around 0.19, but two targeted corrections — removing the lowest-competency quarter of workers using a competency-weighted voting model, and setting aside about 20% of pairs where five judges could not form a majority — raise agreement to roughly 0.48, comparable to established annotation studies. The paper uses these cleaned labels to rank responses per topic with a probabilistic pairwise-comparison model, and reports that LLM responses are significantly preferred over human-written ones on most dimensions while bullet-style responses beat essay and news styles. Against these human-derived rankings, both the LLM-as-judge judgments and reference-based similarity metrics score poorly, which the paper reads as evidence that judgment-based evaluation with crowdsourced pairwise data is the viable path for RAG.","pith_inferences":["A natural next step the paper does not take is to test whether a few-shot LLM judge fine-tuned or prompted with a sample of these human gold labels approaches human-level agreement; the corpus's public release makes that test directly runnable.","Because the gold set excludes the roughly 20% of pairs judges found minimally differentiable, the reported human-versus-LLM quality gap may describe only clearly distinguishable response pairs; in practice, systems whose outputs are all similar in quality might look closer to ties than the headline numbers suggest.","The finding that LLMs write less readable text while humans mirror source readability suggests a cheap, testable intervention: instructing generators to simplify syntax and copy more source-like phrasing could close part of the judged quality gap.","If the reliability result transfers to other topic sets and languages, crowdsourced pairwise utility judgments could become the calibration data for automated RAG metrics, shifting the debate from whether LLMs can replace humans to how much human-labeled data is needed to tune LLM judges."],"forward_implications":["RAG benchmark builders can treat crowdsourced pairwise judgments as a practical source of ground truth, at a cost of roughly six cents per gold judgment once redundancy and filtering are accounted for.","Reference-based evaluation should not be used to rank RAG systems: across utility dimensions, even the strongest tested overlap metric reached only a moderate correlation with the human-derived rankings, and most values were much lower.","Zero-shot LLM judging, at least in the tested configuration, is not a valid substitute for human pairwise judgments, since agreement with gold labels remains poor even when the LLM is highly self-consistent.","Pairwise designs are worth their extra cost over pointwise ones: after the same competency correction, pointwise agreement remained around 0.21, less than half the pairwise level.","Response style is a real factor in perceived quality, so RAG systems that produce bullet-style output currently hold an advantage over essay- and news-style output on these topics."],"supporting_citations":[{"why":"Supplies the seven utility dimensions that structure the judgment study.","marker":"[13]"},{"why":"Provides the pairwise annotation design and the probabilistic pairwise-comparison ranking model used to derive per-topic response grades.","marker":"[14]"},{"why":"Supplies the competency-weighted majority-vote model used to estimate worker reliability and clean the gold labels.","marker":"[18]"},{"why":"Motivates the pairwise design by showing pointwise judgments bias against certain quality dimensions.","marker":"[17]"},{"why":"Defines BLEU, one of the three reference-overlap metrics tested against the human rankings.","marker":"[30]"},{"why":"Defines ROUGE-L, one of the three reference-overlap metrics tested against the human rankings.","marker":"[24]"},{"why":"Defines BERTScore, the embedding-based reference metric tested against the human rankings.","marker":"[49]"},{"why":"Prior evidence that reference-based metrics fail to match human preferences in news summarization, extended here to RAG.","marker":"[50]"},{"why":"Supplies the 301 topics and passage collection used to build the writing and judgment studies.","marker":"[32]"}],"fun_headline_variants":["Human pairwise judgments beat LLM judges for RAG","Crowdsourced pairwise comparisons trump LLM judging","Pairwise human judgments outrank LLM judges in RAG","Best RAG evaluation: crowd pairwise comparisons, not LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reliability conclusion rests on the cleaning step that discards the lowest-competency quarter of workers and about a fifth of response pairs on which five judges could not form a majority; if either exclusion is biased toward or against a response type, the gold labels and every comparison built on them inherit that bias.","fun_headline_variants_meta":{"raw":{"variants":["Human pairwise judgments beat LLM judges for RAG","Crowdsourced pairwise comparisons trump LLM judging","Pairwise human judgments outrank LLM judges in RAG","Best RAG evaluation: crowd pairwise comparisons, not LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1467,"prompt_tokens":986,"completion_tokens":481,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":414}},"tokens_in":602,"tokens_out":481,"duration_ms":4539,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:19:22.678994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the full 47,320 pairwise judgments, recompute per-topic rankings with no competency filtering and no exclusion of minimally differentiable pairs, and compare the resulting system and style orderings to the paper's cleaned rankings; if the human-versus-LLM advantage or the bullet-style advantage disappears under a plain majority vote, then those findings depend on the cleaning step rather than on the raw crowd signal. A complementary check is to have expert judges label the excluded near-tie pairs and test whether the cleaned gold labels agree with experts on exactly those items.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pairwise annotation design and the probabilistic pairwise-comparison ranking model used to derive per-topic response grades."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the competency-weighted majority-vote model used to estimate worker reliability and clean the gold labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the pairwise design by showing pointwise judgments bias against certain quality dimensions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BLEU, one of the three reference-overlap metrics tested against the human rankings."}],"review_version":1}