{"id":"53363452-bd25-4af8-b9e7-1b0303030fde","arxiv_id":"2412.15524","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding human-written reference responses as context improves LLM-as-a-judge agreement with human preferences by up to 3.2%, yielding a new private task-centric benchmark, HREF.","lead":"This paper introduces HREF, a benchmark that uses human-written example answers to help an AI judge decide which chatbot response better follows an instruction. It reports that this human guidance increases agreement with human preferences by up to 3.2% and releases a private test set and leaderboard.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Composite method is selected and evaluated on the same 438-instance set with no held-out human data, so the 69.4% and 3.2% agreement gains may reflect selection bias rather than true generalization.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: method and judge selection are performed on the same 438-instruction set used to report the final agreement rates. This is a classic selection-bias problem, and the absence of any held-out human-annotated set means the paper never tests whether the composite's agreement generalizes. My independent reading confirms this as the most serious threat to the central claim. The proposed check—a held-out split by instruction within each category—directly quantifies the optimism. I also note the additional inconsistency in Section 5.2/Table 6, where the development set is described as unannotated but human agreement rates are reported on it; this does not change the verdict but reinforces the need for guarded interpretation. The paper's direction is plausible and the released data are useful, so a conditional acceptance with a requested validation study remains appropriate.","tokens_in":24415,"tokens_out":5073,"duration_ms":45642,"concrete_test":"Hold out a random 40% of instructions per category from the 438-instance agreement set; select the best method per category on the remaining 60%, and compute the composite's LOO agreement on the held-out 40%. Repeat across 100 random splits. If the mean held-out composite agreement falls at or below human inner agreement (67.0%), or is more than 2 points below the reported 69.4%, the improvement is not robust. Also record how often the full-data per-category choice is not the best on held-out data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central reliability claim rests on Table 2, where the per-category composite method is selected using the same 438-instruction human agreement set (Section 2.1) on which the final 69.4% composite agreement and the 3.2% improvement are then reported. With roughly 40 instructions per category (Figure 5), the differences among candidate methods are within sampling noise, and choosing the maximum per category introduces optimistic selection bias. The paper provides no held-out human preference data: the development set (430) and the private evaluation set (4,258) have no human annotations (Table 4), so the composite's out-of-sample agreement is never measured. Additionally, Section 5.2 and Table 6 report 'Human Agreement Rate' on the development set even though Table 4 marks that set as unannotated, an internal inconsistency that further undermines the reliability evidence. If the in-sample maximum is not a faithful estimate of agreement on new instructions, the headline 'outperforming human annotators by 2.4%' and the benchmark's validity are both weakened.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HREF, a new instruction-following benchmark with 4,258 private instructions across 11 task categories, human-written reference responses, and a composite automatic evaluation that selects the most reliable pairwise evaluation method per category. The authors construct a 438-instruction human agreement set with four preference annotations per instance from a separate annotator group, and use it to compare LLM-as-a-judge, embedding-based, perplexity, and heuristic methods. Their central empirical claims are that including human-written responses as additional context improves agreement with human judges by up to 3.2%, and that the composite method with Llama-3.1-70B-Instruct reaches 69.4% leave-one-out agreement, outperforming the human annotators' inner agreement rate by 2.4%. They also analyze design choices such as judge model, baseline model, prompt template, and evaluation-set size, and they release a public development set, evaluation code, and a private leaderboard.","tokens_in":167,"tokens_out":5499,"duration_ms":64777,"significance":"If the main claims hold, HREF would be a useful community resource: it is the largest task-oriented instruction-following benchmark with a private test set, uses open-weight models as judge and baseline for reproducibility and contamination resistance, and provides per-category results that can guide model development. The experimental design has notable strengths: a diverse model pool of 32 models, explicit length-controlled response sampling, four human annotations per instance, and an annotator group distinct from the response writers. The paper also releases code and data, and its development-set-to-evaluation-set correlation check is a sensible validation step. However, the headline reliability numbers are weakened by the fact that the composite method is selected and evaluated on the same 438-instruction set, so the quantitative claims require additional out-of-sample or bias-corrected evidence before they can be accepted at face value.","major_comments":[{"comment":"The headline composite agreement is an in-sample maximum. The composite method is constructed by selecting, for each of the 11 categories, the evaluation method with the highest agreement in Table 2, and the same 438-instruction human agreement set is then used to report the 69.4% composite agreement and the 2.4% margin over human annotators in Section 3.1. With roughly 40 instructions per category (Figure 5, right), the per-category differences among methods are within sampling noise, and selecting the maximum per category introduces optimistic selection bias. The paper provides no held-out human preference data, no cross-validation estimate, and no confidence intervals for these differences. Since the 4,258-instruction private evaluation set has no human annotations, the reliability of the composite method on new instructions is not directly measured; the authors should add a bias-corrected estimate, such as leave-one-category-out or a small human-annotated validation sample from the evaluation set, before relying on the 3.2% and 2.4% claims.","section":"Section 2.1 and Table 2"},{"comment":"There is an internal inconsistency about whether the development set has human annotations. Table 4 states that the Development set has no human annotations, but Section 5.2 reports 'Human Agreement Rate' for four prompt templates evaluated on the development set. If these rates were computed on the 438-instruction agreement set instead, the text should say so explicitly; if they were computed on the development set, then Table 4 is incorrect. This matters because the prompt-template comparison is used to justify the choice of Prompt A, so the source of the agreement numbers must be unambiguous.","section":"Section 5.2, Table 6 and Table 4"},{"comment":"The reported development-set-to-evaluation-set correlation (Spearman 0.98, Pearson 0.99) is only a consistency check between two automatic evaluation pipelines; it does not validate that the composite method agrees with human judges on the private evaluation set. The authors should clarify this in the text, since otherwise readers may interpret the high correlation as evidence that the reliability established on the 438-instruction agreement set transfers to the larger private set, which is not directly shown.","section":"Section 4.3"}],"minor_comments":[{"comment":"The paper uses 'Human Response-Guided Evaluation' in the title and abstract but introduces the benchmark as 'Human Reference-guided Evaluation of instruction Following (HREF)' in Section 1; please unify the terminology.","section":"Title and Section 1"},{"comment":"The caption says 'Bold numbers are the highest numbers with Llama-3.1-70B-Instruct for each categories'; this phrasing is ambiguous about whether bold marks the best method overall in each category or the best among the Llama-70B methods.","section":"Table 2 caption"},{"comment":"In the sentence 'the strong correlation between our prompt (Prompt D) and AlpacaEval’s prompt (Prompt A)', the prompt labels appear to be swapped relative to Table 6, where Prompt A is the HREF prompt and Prompt D is the AlpacaEval-style prompt.","section":"Section 5.2"},{"comment":"The parsing description says 'a' is mapped to 0 and 'b' or 'tie' to 1; please clarify whether a tie is deliberately treated as a win in the expected-win-rate computation, consistent with Section 4.2, or whether this mapping is only for the agreement-rate calculation.","section":"Appendix C"},{"comment":"There are several typos and grammatical errors, including 'using by using' (Section 1), 'defination' (Appendix B.2), 'poolly' (Section 4.3), 'migrate such risk' (Section 6), and 'Appendex' (Section 4.3 caption); a careful proofreading pass is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about HREF is that the core idea is good and the benchmark will likely be useful, but the headline agreement numbers are not yet trustworthy. The composite method — per-category choice among LLM-as-a-judge with and without human reference, plus RoBERTa embedding — is selected and evaluated on the same 438-instance human agreement set. With roughly 40 instances per category, the differences in Table 2 are within sampling noise, and picking the best method per category on that set inflates the reported 69.4% and the 3.2% improvement. The paper never measures out-of-sample agreement: the development and private evaluation sets have no human annotations, so we only get the in-sample max.\n\nThat said, there is real value here. Using human-written responses as additional context in the judge prompt is genuinely new relative to AlpacaEval's model-generated references, and the finding that human references beat model-generated ones across judge models is credible and useful. The task-centric curation — 11 categories from the InstructGPT taxonomy, 4,258 private instructions — directly addresses the task-skew problem in prior benchmarks, and using open-weight Llama models for both judge and baseline is a sensible step for reproducibility and contamination control. The high correlation between development and private set rankings (0.98 Spearman) is good evidence the benchmark orders models consistently.\n\nThe soft spots are real but not disqualifying. The selection-on-the-same-set problem is the main one; a nested cross-validation or a separately collected human agreement set for method choice would fix it. There are also no error bars on any of the agreement differences, so we can't tell if the 2.4% edge over human inner-agreement is meaningful. I also found an inconsistency: Table 4 says the development set has no human annotations, yet Table 6 reports human agreement rates on the development set. That needs a correction. Finally, the handling of parse failures and ties is under-specified — parse failures are dropped without reporting their rate, and ties count as wins for the target model.\n\nFor whom: anyone building or using instruction-following benchmarks. It deserves a serious referee; the authors have done the work and released the data. But the revision should include a proper holdout validation of the composite selection before the headline claims are taken at face value.\n\nMy recommendation: engage with it, but treat 69.4% and 3.2% as upper-bound estimates until the selection bias is addressed.","headline":"Good benchmark, clever idea; headline agreement gains are selected in-sample and need a held-out check.","tokens_in":25147,"tokens_out":3089,"would_cite":true,"duration_ms":25810,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding a human-written reference to an LLM judge's prompt improves agreement with human preference judgments, and uses that finding to build the HREF benchmark with a private test set and task-level scores.","keywords":["instruction following","LLM-as-a-judge","human reference","evaluation benchmark","pairwise preference","task-centric evaluation","test-set contamination","human agreement"],"falsifier":"Split the 438-instruction human agreement set by category into method-selection and validation halves, or collect a fresh human-annotated set of equal size, and test whether the per-category winners and the 69.4% composite agreement replicate; if a plain Llama-3.1-70B judge without human reference matches or beats the composite on the held-out half, the central reliability claim fails. Alternatively, once human labels for a sample of the private 4,258-instruction set become available, recompute composite agreement on those held-out instances.","tokens_in":24173,"feed_emoji":"⚖️","tokens_out":7286,"duration_ms":54485,"temperature":0.7,"pith_summary":"This paper is trying to establish that automatic evaluation of how well language models follow instructions becomes noticeably more trustworthy when the judge model is given a human-written reference response as additional context. On a set of 438 instructions spanning 11 task categories, the authors find that this added reference raises agreement with human preference judgments by up to 3.2% across a wide range of tasks. They turn this observation into HREF, a benchmark of 4,258 instructions with a private test split, a public development split, a composite per-category evaluation setup, and a live leaderboard. If the finding holds, model developers get a cheaper, faster, and contamination-resistant proxy for human preference judgments, with per-task scores that show which skills to improve.","feed_headline":"Human-written answers boost LLM-judge reliability by 3.2%","feed_subtitle":"New benchmark adds human-written references as judge context and beats human annotators' own agreement by 2.4 points.","key_machinery":"The load-bearing device is the composite evaluation setup. For each of the 11 task categories, the authors compare LLM-as-a-Judge, LLM-as-a-Judge-with-human-reference, and embedding-based similarity to the human reference, and select the method with the highest leave-one-out agreement against four human annotations per instance. The human reference enters the judge prompt as extra context, not as the output to reproduce, and tie judgments are allowed; RoBERTa-Large provides the embedding similarity when that method wins. This per-category selection is what lets HREF report task-level win rates while keeping a single headline reliability number.","core_discovery":"Using human-written responses as reference context, rather than as a gold answer to be matched, improves the reliability of LLM-based judges of instruction following. Across the 438-instruction human agreement set, LLM-as-a-Judge with a human reference outperforms the same judge without it for capable judges, and the composite method that selects the best evaluator per category—Llama-3.1-70B-Instruct as judge, human reference in the prompt, or RoBERTa embedding similarity in open-ended categories—reaches a leave-one-out agreement rate of 69.4% with human annotations. That is 2.4 points higher than the human annotators' own inner-agreement rate of 67.0%. The authors also report that judge models, when directly asked to compare a model response against a human response, strongly prefer the model response, so the human reference should be used as a complementary context rather than as the sole comparison target.","pith_inferences":["Editorial: If the orthogonal-perspective explanation is right, adding more than one human-written reference per instruction, or references written by different human annotators, should push agreement higher still—an extension the paper does not test.","Editorial: The per-category selection is made on the same 438-instruction set whose agreement rates are then reported as the headline result; a holdout-based replication with a fresh human-annotated set would be the natural check of whether the 69.4% figure transfers.","Editorial: Because human annotators themselves agree only 67.0% under leave-one-out, an automatic evaluator has a low ceiling against today's annotation protocol; disagreement-aware aggregation, such as weighting by annotator confidence, might be a more informative target than the mode-based agreement used here.","Editorial: The finding that judges prefer model-written responses when directly compared against human responses suggests reference-guided evaluation could be misused if someone treats the human response as a gold answer; the paper's setup carefully avoids that, and future benchmarks should keep the same distinction."],"forward_implications":["Adding a human-written reference response to LLM judge prompts raises agreement with human preference judgments by up to 3.2% across the evaluated instruction-following tasks.","The per-category composite evaluator using Llama-3.1-70B-Instruct reaches 69.4% leave-one-out agreement with human annotations, 2.4 points above the human annotators' inner agreement rate.","Because HREF uses open-weight models for both baseline and judge and keeps its 4,258-instruction evaluation set private, it avoids both direct test-set leakage and the indirect contamination that comes from sending test prompts to closed API models.","Task-level scores reveal uneven strengths: Mistral-Large-Instruct-2407 has the best overall expected win rate on the initial 37-model run yet performs poorly on Open QA, illustrating the benchmark's stated purpose of surfacing skill-specific gaps.","Model rankings on the public 430-instruction development set correlate strongly (0.98 Spearman) with rankings on the private evaluation set, supporting the use of the development set for fast iteration."],"supporting_citations":[{"why":"Defines the pairwise LLM-as-a-Judge evaluation protocol, including response pairs and preference selection, that HREF re-evaluates.","marker":"Zheng et al. (2023)"},{"why":"Supplies the annotation guidelines, the AlpacaEval prompt comparison, and the leave-one-out agreement methodology used for measuring reliability.","marker":"Li et al. (2023)"},{"why":"Provides the InstructGPT taxonomy of 11 instruction-following task categories that HREF uses as its task-centric structure.","marker":"Ouyang et al. (2022)"},{"why":"No Robots is the source of the instructions and human-written responses for 8 of the 11 task categories and for the development set.","marker":"Rajani et al. (2023)"},{"why":"Defines the Llama-3.1 models used as the judge, baseline, and model pool in HREF.","marker":"Dubey et al. (2024)"},{"why":"WildBench provides the private-test-set precedent and real-user task distribution that HREF contrasts with its task-centric collection.","marker":"Lin et al. (2024)"},{"why":"RoBERTa-Large supplies the embeddings used by the embedding-based evaluator in the composite setup.","marker":"Liu et al. (2019)"}],"fun_headline_variants":["Human references lift LLM judge agreement by 3.2%","New benchmark HREF: human responses guide LLM judges","HREF: human-written references improve LLM evaluation","Beyond gold: human responses as context for LLM judges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The composite method's winning per-category choices and the headline 69.4% agreement are both computed on the same 438-instruction human agreement set, where each category has only about 40 instances, so the claim assumes that these selections, and the agreement rate itself, hold up on the separate 4,258-instruction private evaluation set.","fun_headline_variants_meta":{"raw":{"variants":["Human references lift LLM judge agreement by 3.2%","New benchmark HREF: human responses guide LLM judges","HREF: human-written references improve LLM evaluation","Beyond gold: human responses as context for LLM judges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2469,"prompt_tokens":974,"completion_tokens":1495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1427}},"tokens_in":590,"tokens_out":1495,"duration_ms":10409,"temperature":1.0,"reasoning_tokens":1427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:20:33.494235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split the 438-instruction human agreement set by category into method-selection and validation halves, or collect a fresh human-annotated set of equal size, and test whether the per-category winners and the 69.4% composite agreement replicate; if a plain Llama-3.1-70B judge without human reference matches or beats the composite on the held-out half, the central reliability claim fails. Alternatively, once human labels for a sample of the private 4,258-instruction set become available, recompute composite agreement on those held-out instances.","supporting_citations":[],"review_version":1}