{"id":"b6e2a5d1-d282-464e-b595-34599d02147a","arxiv_id":"2504.20006","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Nugget-based fact-recall scores on Search Arena battles correlate with human preferences, with about 54% agreement, and offer finer-grained diagnostic signals than raw win/loss votes.","lead":"This paper applies an automatic fact-level scoring method, AutoNuggetizer, to about 5,100 single-turn battles from LMArena's Search Arena, showing that differences in atomic-fact recall scores line up with human win/loss votes. The result suggests that nugget-based scoring could make arena-style LLM evaluation more explanatory and diagnostic, though the measured predictive agreement is only about 53-55%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Nuggets are generated from the two candidate responses themselves, so the scoring rubric is not independent of the answers being scored; an ablation that omits responses from nugget generation is needed to rule out a self-fulfilling score difference.","rationale":"The reader's weakest assumption is that GPT-4.1's automatic nuggets and support labels are reliable proxies, with validation inherited from prior TREC studies. I agree that label reliability is the critical premise, but the more specific and more testable problem is that this application changes the nugget-generation source: the two candidate responses are fed into the prompt that creates the nuggets, so the scoring instrument is constructed from the objects being scored. That is not merely inherited-validation uncertainty; it is an identifiable contamination mechanism that can inflate both the KS separation and the confusion-matrix agreement. The paper's own Section 4.3 ablates URL contents while keeping the responses, leaving the opposite ablation undone. A simple re-run without model responses in generation would settle whether the reported alignment is a property of fact recall or an artifact of the pipeline. The remaining issues raised by the reader, such as in-sample threshold selection, lack of a chance baseline, and over-strong language about 54% agreement, are real but secondary: the KS tests do not depend on the threshold, and if the contamination check passes, the paper still needs a held-out threshold and a chance baseline before claiming strong alignment. Because the proposed ablation is straightforward and the central claim is otherwise plausible, conditional acceptance with this additional requirement is the right disposition; hence the reader's verdict is unchanged.","tokens_in":13310,"tokens_out":8250,"duration_ms":91466,"concrete_test":"Rerun the full single-turn pipeline on the subset of battles whose scraped URL content is non-trivial (e.g., more than 100 bytes after scraping, as described in Section 3), generating nuggets from the query and retrieved chunks ONLY, with neither model response included in the AutoNuggetizer generation prompt. Keep assignment, All Score, threshold selection, and analysis identical, using a held-out split or cross-validation for the tie threshold. If the pairwise KS p-values remain below 0.01 and the diagonal agreement rates remain near 54%, the contamination concern is resolved. If the distributional separation or agreement drops materially, the reported alignment is an artifact of constructing the scoring rubric from the scored responses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3, the AutoNuggetizer request for nugget generation includes 'the responses from each model' in addition to the query and retrieved chunks. The nugget set N is therefore not independent of the two responses that are subsequently scored. A statement that appears only in model A's answer, and not in model B's answer or in the retrieved corpus, can be emitted as a nugget; the assignment step then credits A with supporting it. The score difference d = score_B - score_A thus partly measures which model's content the nuggetizer chose to treat as required content, rather than recall of facts that a good answer must contain independently. This contamination can inflate the KS distributional separation in Figure 3 and the diagonal agreement in Figure 4: a response with more unique content contributes more nuggets to N and also supports those nuggets. Section 4.3 ablates URL contents but never removes the candidate responses from the nugget-generation prompt, so the direction and size of this bias are untested. Validation of AutoNuggetizer on TREC data (ref [23]) does not transfer cleanly because in those evaluations the generation source is not the test responses. This is the weakest load-bearing link in the claim that nugget scores explain human preferences.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies the AutoNuggetizer framework to 5,103 single-turn battles from LMArena's Search Arena v1 dataset. For each battle, it generates information nuggets from the query, scraped URL contents, and the two model responses; assigns support labels to each response; computes a nugget score; and compares the score difference with the human win/tie judgment. The authors report that the three conditional distributions of nugget score differences are significantly different (pairwise KS p-values ≤ 1.2e−24), that a confusion-matrix comparison with a tie threshold of 0.07 yields 54.7%/52.5% agreement on model-A/model-B wins, that inversion rates vary across query classes and languages, and that using only model responses for nugget generation gives comparable agreement. They also benchmark an LLM-as-a-judge baseline, which agrees more on clear wins but rarely predicts ties. The paper argues that these results demonstrate the explanatory and diagnostic value of nugget-based evaluation.","tokens_in":13561,"tokens_out":3927,"duration_ms":36556,"significance":"If the headline results are taken at face value, the paper offers a scalable, fully automatic way to turn preference judgments into component-level explanations: nugget recall differences partially track human win/tie decisions, and the per-nugget support labels could diagnose retrieval versus generation failures. The release of code and the use of public LMArena data are strengths, as is the explicit comparison to an LLM-as-a-judge baseline. However, the central quantitative claims—the agreement rates and 'strong alignment'—rest on an in-sample tuned tie threshold, a post hoc metric choice, and a nugget-generation step that includes the very responses being scored. These issues must be resolved before the explanatory-power claim is fully supported.","major_comments":[{"comment":"The tie threshold of 0.07 is selected by sweeping values between 0.05 and 0.15 on the same human labels used to compute the agreement rates: 'This threshold was selected by sweeping values between 0.05 and 0.15 in increments of 0.01... to most closely reflect an equal distribution of modelA wins, modelB wins, and ties when the human preference is a tie.' This is in-sample tuning; the reported diagonal percentages (54.7% and 52.5%) are therefore optimistic estimates, and no confidence intervals are provided. Please report the agreement rates across the full threshold sweep, use held-out data or a pre-specified threshold, and include bootstrap or exact binomial confidence intervals for the reported agreement rates.","section":"Section 4, Figure 4"},{"comment":"The AutoNuggetizer prompt includes 'the responses from each model' in addition to the query and retrieved chunks. Because the nugget set is generated from the two candidate answers, the subsequent assignment step credits each model for content that the generator selected partly because it appeared in that model's response. This makes the score difference d = score_B − score_A a partially self-referential quantity, not a pure measure of recall of facts a good answer must contain. Section 4.3 ablates URL contents but never removes the candidate responses from the generation prompt, so the direction and size of this bias are unknown. Please run an ablation that generates nuggets from the query and retrieved chunks only (with a fallback for battles with unusable URLs) and compare the KS results and agreement rates.","section":"Section 3, Nugget generation"},{"comment":"The choice of 'All Score' over 'Strict Vital' and the other two combinations is justified post hoc on the same data: 'we adopt the \"All Score\" metric, which achieves the highest recall... the \"Strict Vital\" metric... is too strict for our use case.' This metric selection, like the tie-threshold tuning, uses the human votes that are later used to measure agreement. Please report results for all four metric combinations, or justify the metric a priori, so that the reported alignment is not a product of selection.","section":"Section 3, Nugget evaluation"},{"comment":"The text describes the results as 'strong alignment' and 'strongly correlated' (Section 5) and 'demonstrate a strong alignment' (Section 7), yet the headline agreement rates are 54.7% and 52.5% for clear winners, with lower rates for ties, and no correlation coefficient is computed. Please qualify these claims and include a comparison to a trivial baseline (e.g., always predicting the majority class) to calibrate what these agreement rates mean.","section":"Sections 5 and 7"}],"minor_comments":[{"comment":"Figure 1 contains typos: 'Answer Reponse' and 'longetivity' in the displayed response text.","section":"Figure 1"},{"comment":"The KDE bandwidth of 0.5 is not justified, and the axis labels repeat 'Density' without a clear quantity; a brief note on bandwidth selection would improve reproducibility.","section":"Figure 2"},{"comment":"Battles where both responses are 'Tie (both bad)' are excluded from the distributions in Figure 2 but are included in the confusion matrices in Figures 4 and 8; please clarify the treatment of this category consistently across analyses.","section":"Section 4, Figure 2 vs Figure 4"},{"comment":"The statement 'assuming a similar distribution of query categories across languages' is made without supporting evidence; since language-specific query classification is deferred, this assumption should be flagged more explicitly as an untested assumption.","section":"Section 4.2"},{"comment":"The venue description for Rackauckas et al. appears garbled: 'co-located with 10th International Conference on Online Publishing (SIGIR 2024)' is not a recognizable conference name; please correct the reference.","section":"Reference [25]"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.IR and has a useful, reproducible pipeline, but the central agreement numbers are not yet trustworthy because of in-sample threshold/metric selection and the self-referential nugget generation. The KS distributional finding may survive the ablation, but the agreement rates and 'strong alignment' language need rework. I would encourage the editor to ask for the response-free nugget generation ablation and threshold robustness analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is a legitimate extension of the AutoNuggetizer framework to Search Arena, and the main results are plausible but somewhat oversold. The nugget method itself is prior work from the same group; what is new is the application to roughly 7K Search Arena battles, plus the breakdowns by query type and language and the comparison against an LLM judge. Those are real additions, and the code is public.\n\nWhat it does well: the pipeline is described transparently, the exclusion of multi-turn battles is principled, and the KS tests do show genuine distributional differences across human vote categories. The query-class analysis is the most useful part: inversions concentrate in ambiguous, assumptive, and multi-faceted queries, which matches intuition, and the LLM-judge comparison is a fair baseline that shows a real trade-off.\n\nSoft spots: the \"strong alignment\" language outruns the numbers. Agreement on clear wins is around 54%, which is above chance but not strong. The 0.07 tie threshold was swept in-sample on the same battles used to measure agreement, and the All Score metric was chosen post hoc. No confidence intervals are reported, so the diagonal is likely an optimistic estimate. The stress-test note also lands: nuggets are generated from the two responses themselves, so the scoring rubric is not independent of the answers being scored. A unique fact in model A's response can become a nugget, and A then gets credit for it. Section 4.3 ablates URL contents but never removes the responses from the generation prompt, so the direction and size of the contamination are untested. The TREC validation does not transfer cleanly because there the nuggets come from reference pools, not from the outputs under test. This is a load-bearing assumption, but it is testable: generate nuggets from query and chunks only and see if the distributional separation survives.\n\nBottom line: this paper deserves a serious referee. I would ask for the response-free nugget ablation, held-out threshold selection, and softer claims before accepting. The core finding is plausible but needs that check before it is used to interpret leaderboard scores.","headline":"A useful, honest extension of nugget-based evaluation to Search Arena, with a real but addressable circularity concern in nugget generation.","tokens_in":14045,"tokens_out":2385,"would_cite":true,"duration_ms":25792,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated nugget scoring of two search-augmented LLM answers separates human win, tie, and loss verdicts: the three score-difference distributions differ at $p \\leq 1.2 \\times 10^{-24}$.","keywords":["nugget-based evaluation","human preference","LLM evaluation","retrieval-augmented generation","search arena","automatic fact recall","Kolmogorov-Smirnov test","diagnostic evaluation"],"falsifier":"Take a random sample of, say, 300 single-turn battles, have two human annotators independently list the atomic facts a good answer should contain and score the same two model responses, then compare those human-nugget score differences across the three human-vote groups; if the separation disappears or if the automatic support labels disagree with human labels on more than a small fraction of nuggets, the central correlation is an artifact of the automatic nuggetizer rather than a property of answer content.","tokens_in":13134,"feed_emoji":"⚖️","tokens_out":7443,"duration_ms":71957,"temperature":0.7,"pith_summary":"Battles, or side-by-side comparisons where humans pick the better LLM answer, are widely used but tell you only who won, not why. This paper tests whether automatic fact-recall scoring can explain and diagnose those judgments. On roughly 5,100 single-turn battles from the public Search Arena data, it splits each answer into atomic facts, asks a language model which facts a good answer should contain, and scores each answer by how many of those facts it supports. The paper finds that the difference in these scores lines up with human verdicts: the score-difference distributions for A-wins, ties, and B-wins are distinct at $p$-values below $1.2 \\times 10^{-24}$, and a simple threshold agrees with the human winner in about 52--55% of decisive battles. The payoff is an evaluation signal that says what fact coverage drove the win and which query types (ambiguous, assumptive, multi-faceted) resist fact-based scoring.","feed_headline":"Nugget scores align with human LLM battle votes","feed_subtitle":"Splitting answers into atomic facts predicts who wins, who loses, and who ties in search-augmented AI comparisons.","key_machinery":"The central machinery is the automatic nuggetization pipeline, which (1) uses an LLM to generate atomic facts, or nuggets, from the query, the top retrieved chunks, and the two candidate responses, labeling each nugget vital or okay; (2) assigns each response a support label of support, partial support, or no support for every nugget; and (3) aggregates an All Score that counts nuggets of all importance and support levels. The paper then feeds the per-battle score differences into kernel density estimates, pairwise Kolmogorov-Smirnov tests, and a confusion matrix with a 0.07 tie threshold to relate the score to human verdicts. That pipeline is what converts raw preference judgments into a structured, per-fact explanation of why one answer won.","core_discovery":"Conditioned on the human verdict, the distribution of nugget score difference $score_B - score_A$ is not the same across the three outcomes: it shifts left when model A wins, centers near zero on ties, and shifts right when model B wins. Pairwise Kolmogorov-Smirnov tests give statistics 0.205 to 0.313 with $p$-values at most $1.2 \\times 10^{-24}$, so the paper concludes that nugget score differences align with human preferences in head-to-head evaluation of search-augmented LLMs. With a tie threshold of 0.07 on the score difference, the automatic preference matches the human winner in 54.7% of model-A-winning battles and 52.5% of model-B-winning battles; disagreements, called preference inversions, concentrate in ambiguous, assumptive, and multi-faceted queries, with German-language battles showing the highest inversion rate. The paper also reports that generating nuggets from the two LLM responses alone, without the scraped URL contents, gives essentially the same agreement with human preferences as the full pipeline.","pith_inferences":["The reported roughly 50--55% agreement understates the signal in the score; with calibrated thresholds or a proper scoring rule, the nugget score difference could be converted into a probabilistic predictor of human choice and compared against other predictors explicitly.","Because the 0.07 tie threshold and the distribution separation are fit on this dataset, applying the same nuggetization to a new arena will likely require its own threshold calibration, otherwise the apparent alignment may shrink.","The inversion pattern suggests a possible two-stage evaluation: use nugget scores for knowledge-intensive queries, which show the highest alignment, and fall back to human judgment or a different metric for ambiguous and multi-faceted queries.","The finding that responses-only nuggetization preserves agreement implies the method could be applied retroactively to standard arena battles that lack retrieved documents, turning historical preference data into a diagnostic signal."],"forward_implications":["Automated nugget scoring can serve as an explanatory overlay on arena battle data, showing which answer contained more of the facts a good answer needs.","The method is diagnostic: inversion rates identify query types and languages where human and nugget preferences diverge, pointing to where evaluation of RAG systems needs refinement.","Because nuggetization from LLM responses alone matches the version that uses scraped URL contents, the approach transfers to settings without reliable external evidence.","Compared with an LLM-as-a-judge baseline, nugget scoring produces fewer preference inversions and does not collapse the tie category, so it offers a structured complement to direct judging.","The score-difference distributions provide a principled basis for setting tie thresholds, since the threshold can be tuned to match the observed human tie rate."],"supporting_citations":[{"why":"Supplies the automatic nugget generation and assignment pipeline used end-to-end here.","marker":"[23]"},{"why":"Defines the All Score metric and the TREC 2024 RAG validation that the paper inherits for nugget scoring.","marker":"[24]"},{"why":"Provides the Search Arena V1 battle data, including human vote categories and URLs, that all experiments use.","marker":"[18]"},{"why":"Establishes the arena side-by-side human-preference evaluation paradigm that battles instantiate.","marker":"[4]"},{"why":"Provides the eight-attribute query rating rubric used to classify queries and locate preference inversions.","marker":"[26]"},{"why":"Contributes the chain-of-thought pairwise judging prompt adapted for the LLM-as-a-judge baseline.","marker":"[25]"},{"why":"Introduces the original nugget-based evaluation methodology for question answering that this pipeline operationalizes.","marker":"[29]"}],"fun_headline_variants":["Nugget scores side with human LLM battle verdicts","Atomic facts explain who wins LLM arena duels","AutoNuggetizer matches human preferences in 7K battles","URLs unnecessary: nuggets alone align with human LLM votes","Nugget score gaps mirror human winners in RAG duels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the automatically extracted nuggets are a faithful, unbiased inventory of the facts a good answer must contain, without manual verification on this dataset, so if those labels are wrong or biased toward the scoring model's own style, the observed alignment with human preferences could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Nugget scores side with human LLM battle verdicts","Atomic facts explain who wins LLM arena duels","AutoNuggetizer matches human preferences in 7K battles","URLs unnecessary: nuggets alone align with human LLM votes","Nugget score gaps mirror human winners in RAG duels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000314,"raw_usage":{"total_tokens":1796,"prompt_tokens":970,"completion_tokens":826,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":739}},"tokens_in":586,"tokens_out":826,"duration_ms":8225,"temperature":1.0,"reasoning_tokens":739,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:36:54.812445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 300 single-turn battles, have two human annotators independently list the atomic facts a good answer should contain and score the same two model responses, then compare those human-nugget score differences across the three human-vote groups; if the separation disappears or if the automatic support labels disagree with human labels on more than a small fraction of nuggets, the central correlation is an artifact of the automatic nuggetizer rather than a property of answer content.","supporting_citations":[{"cited_title":"Angelopoulos, Wei-Lin Chiang, Narges Norouzi, and Joseph E","cited_arxiv_id":null,"evidence_quote":"Provides the Search Arena V1 battle data, including human vote categories and URLs, that all experiments use."},{"cited_title":"Chatbot arena: An open platform for evaluating LLMs by human preference","cited_arxiv_id":null,"evidence_quote":"Establishes the arena side-by-side human-preference evaluation paradigm that battles instantiate."},{"cited_title":"Evaluating rag-fusion with ragelo: an automated elo-based framework","cited_arxiv_id":null,"evidence_quote":"Contributes the chain-of-thought pairwise judging prompt adapted for the LLM-as-a-judge baseline."},{"cited_title":"V oorhees","cited_arxiv_id":null,"evidence_quote":"Introduces the original nugget-based evaluation methodology for question answering that this pipeline operationalizes."}],"review_version":1}