{"id":"14bf87db-6dd7-4e14-8a7e-1befb7ee1fb4","arxiv_id":"2501.00257","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"EQUATOR uses vector search to fetch a human reference answer and a binary LLM grader to score open-ended LLM responses, producing far lower scores than standard benchmarks.","lead":"This paper introduces EQUATOR, a framework that grades open-ended LLM answers by retrieving a similar human-evaluated reference answer and asking a small language model to check for an exact match. The authors report that EQUATOR gives models much lower scores than traditional methods, and they take this as evidence that traditional evaluation overrates LLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix 15.1's own evaluation table shows EQUATOR scoring demonstrably correct answers as 0 (Q7, Q21), so the reported score drops may be evaluator false negatives, not penalized factual inaccuracies.","rationale":"The reader's weakest assumption—that the human-evaluated answer keys are correct and complete ground truth—is closely related, but the concrete failures in Appendix 15.1 show the problem is even more direct: Q7's answer key is correct and matches the student's answer, yet the LLM evaluator still returns 0 with a contradictory explanation. Q21 shows the answer-key set, or the evaluator's semantic judgment, rejects a fully valid answer. Thus the load-bearing flaw is not only key completeness but the reliability of the matching step itself. The paper provides no external validation against human grading, and the only per-item evidence it includes already contradicts the scoring. A single concrete counterexample is sufficient to invalidate the claim that the score drops are accurate measures of factual inaccuracy. The model-level paired t-tests in Sections 5.1 and 7 (n=13 and n=9) cannot repair this validity gap, since they only show that the two scoring methods differ, not that EQUATOR's scores are correct. Because the central claim is unsupported, the reader's REJECT verdict is appropriate and no adjustment is needed.","tokens_in":28224,"tokens_out":6508,"duration_ms":65968,"concrete_test":"Re-score the 30 proof-of-concept items in Appendix 15.1 with two independent human raters using the published human answer keys and a binary correct/incorrect rubric, with pre-specified equivalence rules. Compare each EQUATOR score to the raters' consensus. If both raters mark Q7 ('Bob') and Q21 ('Astonishing aardvarks adore afternoon activities') as correct while EQUATOR gives 0, the false-negative rate is nonzero and the reported effect sizes conflate evaluator error with model inaccuracy. To separate cause, re-run those two items through the EQUATOR pipeline ten times; stable 0 scores would show the failure is systematic, not a one-off sampling artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that EQUATOR 'more accurately penalizes factual inaccuracies' and that its large score drops (d=1.07, d=2.85) reveal LLM reasoning failures—rests on the assumption that the 0/100 score reflects whether the student answer is actually correct. Appendix 15.1's proof-of-concept table directly undercuts that assumption. For Q7, the student answer is 'Bob'; the human answer key is 'Bob is on Alan's immediate right because it is stated that Alan is on Bob's immediate left.' EQUATOR returns score 0 with the explanation that Emily must be on Alan's immediate right. For Q21, the student's 'Astonishing aardvarks adore afternoon activities' satisfies the task ('every word starts with the letter A'), yet EQUATOR returns 0, asserting those words start with letters other than A. These failures are not ambiguous edge cases: they are false negatives produced by the same LLM-evaluator-plus-single-answer-key mechanism used for the headline experiments. Consequently, the lower mean scores and huge effect sizes cannot be attributed to 'penalizing factual inaccuracies'; they may simply reflect evaluator error and a rigid rubric that rejects acceptable answers. Further, the framework is not deterministic: the final score depends on a stochastic LLM's interpretation of the retrieved key, and the reported paired t-tests are computed on model-level mean differences (n=13; n=9) rather than on paired question-level responses, so statistical significance does not establish measurement validity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes EQUATOR, a framework that pairs open-ended questions with human-evaluated answers stored in a vector database, retrieves the closest answer key via cosine similarity, and uses a locally hosted LLaMA 3.2B model as a strict binary (0 or 100) scorer. The authors apply EQUATOR to two Williams and Huckle benchmarks, report significantly lower mean scores than the original human/LLM scoring (Cohen's d = 1.07 and d = 2.85), and interpret this as evidence that traditional methods overestimate LLM factual accuracy. A 30-question proof-of-concept evaluation is included in Appendix 15.1.","tokens_in":28488,"tokens_out":6661,"duration_ms":60053,"significance":"The problem of evaluating open-ended LLM reasoning without fluency bias is important, and the paper deserves credit for attempting a transparent, locally hosted evaluation pipeline and for candidly discussing sources of bias in Section 12. However, the empirical contribution does not establish the central claim. The proof-of-concept table in Appendix 15.1 contains clear false negatives produced by the identical scoring mechanism used in the headline experiments, the framework is not deterministic in the sense claimed, and the statistical analyses are performed on model-level means with very small samples. If the false-negative rate were corrected and the statistical claims were re-run on question-level data, the framework could become a useful engineering contribution; as presented, the reported score drops and effect sizes cannot be interpreted as evidence about factual accuracy.","major_comments":[{"comment":"The proof-of-concept table directly contradicts the paper's interpretation of 0/100 scores. For Q7, the student answer 'Bob' is exactly the answer given in the human key ('Bob is on Alan's immediate right...'), yet EQUATOR returns 0 with an explanation that contradicts its own key. For Q21, the student sentence 'Astonishing aardvarks adore afternoon activities' satisfies the constraint that every word starts with A, yet the evaluator returns 0 and says the words start with other letters. These are not ambiguous edge cases: the same retrieval-plus-LLM-judge mechanism is used for Experiments 1 and 2. The lower means and large effect sizes (d=1.07, d=2.85) may therefore be artifacts of evaluator false negatives rather than evidence of 'more accurately penalizing factual inaccuracies' as claimed in Sections 5.1.2 and 8.1.1.","section":"Appendix 15.1, Q7 and Q21"},{"comment":"The formal scoring definition in Step 3 uses a similarity threshold tau: Match(Ae, Kbest) = 1 if Sim(Ae, Kbest) >= tau, and 0 otherwise. Section 3.3, item 5, states 'Instead of setting an empirical similarity threshold, we retrieve the most semantically similar answer from the vector database as the answer key,' and no value of tau is reported anywhere. This is an internal inconsistency in the core algorithm, and tau is a free parameter that should either be specified or removed from the formal definition.","section":"Section 3.4, Step 3"},{"comment":"The paired t-tests are computed on model-level mean scores (n=13 in Experiment 1, n=9 in Experiment 2), not on paired question-level responses. This violates the pairing assumption stated in Section 5.1 ('the same models evaluated on the same questions'), and the Central Limit Theorem justification is not credible for n=9 or n=13. The Limitations section itself says '30 questions are not enough to achieve statistical significance,' directly contradicting the paper's earlier claims of statistical significance (p<0.005 and p<0.0001). The reported p-values establish only that the average difference across models is nonzero, not that EQUATOR measures student-answer correctness more accurately.","section":"Sections 5.1, 6, and Section 12"},{"comment":"The evaluator prompt contains contradictory scoring instructions: it first says 'If it does, assign a score of 100%. Otherwise, you must assign a score of 0%,' but later says 'If the Answer Key matches the student's answer, you must assign a score of 0%; no partial credit is allowed.' A prompt with internal contradictions cannot support the paper's claim of deterministic, reproducible scoring, and it may contribute to the false negatives documented in Appendix 15.1.","section":"Section 3.5.2"},{"comment":"The framework is not deterministic. The final score depends on the stochastic output of LLaMA 3.2B interpreting the retrieved answer key, and on the cosine retrieval of the closest key; no temperature, seed, or sampling parameters are reported. The word 'deterministic' appears throughout but is never justified by controlling randomness or by showing run-to-run agreement.","section":"Sections 3.2 and 3.5"},{"comment":"Table 2 (Experiment 1, multiple-choice benchmark) and Table 5 (Experiment 2, open-ended benchmark) report identical means, standard deviations, and confidence intervals for the same model names. Since the two experiments are supposed to use different datasets, this duplication means the results of Experiment 2 cannot be verified as reported.","section":"Table 2 and Table 5"}],"minor_comments":[{"comment":"The abstract cites 'Williams and Huckle (2004)' while the rest of the paper correctly uses 2024; please correct the citation.","section":"Abstract"},{"comment":"The heading reads '2014-6-12-Benchmarks' but should read '2024-06-12-Benchmark.'","section":"Section 6 heading"},{"comment":"There is a typo in 'Toxonmony' and a formatting error in '15.1Our'; both should be corrected.","section":"Section 3.6"},{"comment":"The paragraph beginning 'In our analysis, the EQUATOR Evaluator demonstrated significant token savings...' is repeated verbatim before and after Table 9; one copy should be removed.","section":"Section 10"},{"comment":"Several figure captions in the Experiment 2 appendix reuse '9-13 Multiple-Choice Benchmark' labels (e.g., Figures 10, 11, and 13), which makes the appendix confusing and should be corrected to refer to the 6-12 open-ended benchmark.","section":"Section 15.5"},{"comment":"The question numbering is inconsistent: item 18 repeats the poisoned-wine question already shown as item 17, and there are two items numbered 30.","section":"Appendix 15.1"},{"comment":"The model names are inconsistent across the paper: Section 3.2 uses 'll-minilm,' Section 3.3 mentions LLaMA 3.2B, and Section 10 refers to 'Ollama 3.2'; the exact embedding and evaluator models should be named consistently.","section":"Sections 3.2, 3.3, and 10"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early draft with duplicated tables, inconsistent numbering, and self-contradictory statements. The absence of code or data release, combined with the unaddressed false negatives in the paper's own appendix, makes the empirical claims unverifiable. The central claim about more accurate penalization of factual inaccuracies is not supported by the evidence, and the statistical analysis does not provide a valid basis for the reported effect sizes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the framework is a reasonable idea, but the central evidence for it does not survive contact with its own appendix. Appendix 15.1 shows the evaluator scoring demonstrably correct answers as 0 (Q7: 'Bob' is the correct answer, scored 0; Q21: the all-A sentence is valid, scored 0). So the large score drops and Cohen's d values cannot be read as evidence that EQUATOR 'more accurately penalizes factual inaccuracies'—they may just be false negatives from the LLM judge plus a binary rubric tied to a single answer key.\n\nWhat's new: the specific configuration—retrieve the nearest human answer by cosine similarity, then binary-match with a local LLM—is a combination of known pieces, but the empirical score tables on the Williams & Huckle benchmarks are new. The paper also does some things well: it is transparent about the prompts, includes a detailed appendix with worked examples, and flags limitations (30 questions, clustering, bias sources). That transparency is what lets us see the flaw.\n\nSoft spots: the central claim is unsupported. No validation against human grading is provided; the statistical tests are computed on model-level mean differences (n=13, n=9) rather than paired question-level responses, so they don't establish measurement validity. The 'deterministic' label contradicts the stochastic LLM evaluator. The formal definition includes a threshold tau that the text later says is not used. And the paper's own limitations section admits 30 questions aren't enough for statistical significance, yet the abstract and conclusions still claim a paradigm shift. These are not minor quibbles; they affect the interpretation of every reported number.\n\nVerdict: this is a promising direction but the current manuscript would not survive peer review as is. The authors would need to validate the evaluator against human judgments, fix the statistics, and address the false-negative examples. A serious editor should desk reject this version—it's a draft, not a paper. If the authors come back with those fixes, it's worth another look.","headline":"A promising evaluation idea undermined by its own appendix: the proof-of-concept shows false negatives on obviously correct answers, so the headline score drops don't support the paper's claims.","tokens_in":29036,"tokens_out":2613,"would_cite":false,"duration_ms":26696,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that EQUATOR, a deterministic scoring framework that grades open-ended LLM answers against human-evaluated answer keys in a vector database, yields significantly lower and more accurate scores than traditional…","keywords":["LLM evaluation","open-ended questions","deterministic scoring","vector database","cosine similarity","binary grading","factual accuracy","reasoning benchmarks"],"falsifier":"Run EQUATOR on a set of open-ended questions where each answer key has several widely accepted correct wordings that differ only in phrasing. If a correct paraphrase that is not the exact stored string consistently receives 0, then the binary answer-key rule is penalizing surface mismatch rather than factual error, and the central claim would fail.","tokens_in":28010,"feed_emoji":"🎯","tokens_out":9146,"duration_ms":75662,"temperature":0.7,"pith_summary":"The paper proposes EQUATOR, a deterministic scoring framework for open-ended reasoning questions. It stores human-evaluated answer keys in a vector database, retrieves the closest key for each student answer by cosine similarity, and uses a small local language model to assign 100% only when the final answer matches the key and 0% otherwise. Comparing this scoring with the published human and multiple-choice scores on two benchmark sets, the paper finds that EQUATOR gives significantly lower mean scores, with standardized effect sizes of $d = 1.07$ and $d = 2.85$. The authors interpret this as evidence that traditional evaluations overestimate LLM factual accuracy, and they report that EQUATOR also reduces reliance on human graders and cuts token use by 12.20% (22.61% when run locally). If the interpretation is right, EQUATOR offers a scalable and reproducible way to expose where LLMs fail at reasoning.","feed_headline":"LLM reasoning scores drop sharply under strict answer-key grading","feed_subtitle":"Open-ended answers graded against human keys reveal large gaps that multiple-choice tests miss.","key_machinery":"The load-bearing mechanism is a retrieval-and-match pipeline: questions and human-evaluated answers are embedded into 384-dimensional vectors and stored in a vector database; a student answer is embedded the same way; the database returns the answer key with the highest cosine similarity to the student answer; and a prompt-constrained local LLM (the evaluator) returns a score of 100 if the student's final answer matches that key and 0 otherwise. The binary scoring rule is what does the work: it removes threshold tuning, partial credit, and any credit for reasoning or eloquence, so the score reflects only whether the final answer agrees with the chosen reference answer.","core_discovery":"On its own terms, the paper establishes a new way of measuring LLM reasoning: instead of selecting among options or having a human judge weigh reasoning quality, every open-ended answer is compared, through an embedding search, to a small set of human-written correct answers and then graded by a strict binary rule. Applied to the 2024-09-13 multiple-choice benchmark, EQUATOR lowered mean scores by 11.59 points on average relative to the published scoring, with $p \\approx 0.0026$ and $d \\approx 1.07$. Applied to the 2024-06-12 open-ended benchmark, the drop was 20.38 points, with $p < 0.0001$ and $d \\approx 2.85$. The paper claims these large gaps mean that conventional scoring methods award credit for fluency, reasoning, and partial credit, while EQUATOR penalizes factual mismatch even when the answer is persuasive. It further claims that a smaller locally hosted LLM can carry out the grading consistently, eliminating the need for human scoring of each response.","pith_inferences":["The paper does not test whether its 0/100 rule distinguishes 'not matching the stored key' from 'factually wrong'; a natural next experiment is to have independent human graders mark the same responses and measure disagreement.","Because the retrieval step depends on the embedding model, the robustness of the score gaps could be probed by repeating the evaluation with different sentence-embedding models; if the gaps vanish, part of the effect is an artifact of the embedding choice rather than the scoring rule.","A testable extension is to keep the answer-key anchor but allow graded partial credit (for example, 100/70/40/0) and see whether model rankings change; this would tell developers how much information the binary rule discards.","The paper discusses high-stakes deployment in medicine, law, and finance but does not test it; the same answer-key-plus-vector-database design would need validation on those domains where the cost of a fluent wrong answer is high."],"forward_implications":["Same models and same questions receive substantially lower scores under EQUATOR than under published human or multiple-choice scoring, so standard benchmarks likely overstate factual accuracy on reasoning tasks.","Open-ended evaluation no longer requires a human grader for every response; a small local LLM and a vector database of vetted answers can produce reproducible scores.","The framework's reported token savings (12.20% overall, 22.61% with local evaluation) make large-scale evaluation cheaper than sending full multiple-choice prompts to an API.","Because persuasive but wrong answers receive 0, EQUATOR isolates factual failure points that fluency-biased evaluations hide.","The larger score gap on the open-ended benchmark ($d = 2.85$) than on the multiple-choice benchmark ($d = 1.07$) suggests that open-ended scoring captures more of what multiple-choice misses."],"supporting_citations":[{"why":"It supplies the two benchmark datasets, the question taxonomy, and the human-scored answer keys that EQUATOR embeds in its vector database and re-scores.","marker":"[1]"},{"why":"It provides the evidence that multiple-choice evaluation is efficient and robust, the baseline that EQUATOR argues misses open-ended factual errors.","marker":"[13]"},{"why":"It supplies the sentence-embedding method used to turn questions, student answers, and answer keys into vectors for cosine similarity retrieval.","marker":"[14]"},{"why":"It supports the use of a large language model as an automated evaluator in place of human graders.","marker":"[8]"},{"why":"It establishes the standardized effect-size statistic and its interpretation used to quantify the significance of the score differences.","marker":"[22]"},{"why":"It documents likelihood-based evaluation bias in LLM graders, which the deterministic design is intended to avoid.","marker":"[3]"}],"fun_headline_variants":["Strict answer keys cut LLM reasoning scores by 20+ points","LLM reasoning scores plummet with deterministic grading","Open-ended grading reveals large LLM score gaps","Strict grading exposes LLM reasoning inflation vs MC tests","Answer-key grading shows LLMs overcrediting open-ended reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the human-evaluated answer keys stored in the vector database are correct and complete; if any key is wrong, ambiguous, or missing an acceptable alternative, EQUATOR will mark a correct student answer as zero.","fun_headline_variants_meta":{"raw":{"variants":["Strict answer keys cut LLM reasoning scores by 20+ points","LLM reasoning scores plummet with deterministic grading","Open-ended grading reveals large LLM score gaps","Strict grading exposes LLM reasoning inflation vs MC tests","Answer-key grading shows LLMs overcrediting open-ended reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00081,"raw_usage":{"total_tokens":3603,"prompt_tokens":1044,"completion_tokens":2559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":2479}},"tokens_in":660,"tokens_out":2559,"duration_ms":26684,"temperature":1.0,"reasoning_tokens":2479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:55:14.522610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EQUATOR on a set of open-ended questions where each answer key has several widely accepted correct wordings that differ only in phrasing. If a correct paraphrase that is not the exact stored string consistently receives 0, then the binary answer-key rule is penalizing surface mismatch rather than factual error, and the central claim would fail.","supporting_citations":[{"cited_title":"Easy problems that llms get wrong","cited_arxiv_id":null,"evidence_quote":"It supplies the two benchmark datasets, the question taxonomy, and the human-scored answer keys that EQUATOR embeds in its vector database and re-scores."},{"cited_title":"Cohen, Statistical Power Analysis for the Behavioral Sciences","cited_arxiv_id":null,"evidence_quote":"It establishes the standardized effect-size statistic and its interpretation used to quantify the significance of the score differences."}],"review_version":1}