{"id":"e54747c5-bb2e-4486-83ef-029305ff11e9","arxiv_id":"2607.22949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On the MAVE benchmark, switching from exact to fuzzy matching changes reported F1 by ~0.12, dwarfing model choice (~0.005) and prompt choice (~0.024), and fuzzy-match auditing labels 23.2% of exact-match failures as semantically correct.","lead":"Grading the same AI outputs with fuzzy instead of exact matching changes reported accuracy by ~12 points, while swapping models changes it by ~0.5 points. The study says this measurement choice, not the model, dominates what practitioners see.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 23.2% 'ground truth noise rate' is not established because the fuzzy-match proxy for semantic correctness (rapidfuzz partial ratio, τ=80) admits false positives, e.g. substring matches that are incomplete or imprecise; the 50-case manual audit is too small to validate all 737 fuzzy successes.","rationale":"The reader's weakest_assumption already identified the same load-bearing concern: the reliance on rapidfuzz partial-ratio similarity at τ=80 as a proxy for semantic correctness. The paper's own examples and definition show that partial-ratio matching can count substring or unit-mismatch predictions as correct, so the 23.2% 'noise rate' is not a trustworthy estimate of ground-truth noise without exhaustive or properly sampled human validation. The variance-dominance claim (∆F1 = 0.121 vs 0.005) is a separate empirical observation about F1 ranges and is less affected, but the interpretation that this gap is due to benchmark noise rather than merely metric permissiveness depends directly on the noise audit. Therefore the reader's CONDITIONAL verdict is appropriate: the paper's direction is plausible, but the headline noise-rate claim requires stronger validation. Our read does not move the verdict.","tokens_in":11190,"tokens_out":4392,"duration_ms":43451,"concrete_test":"Manually annotate a random sample of all 737 fuzzy-success cases (not just 50 'representative' ones) by two independent raters, blinded to the fuzzy score, with a third adjudicator. Compute the inter-annotator agreement and the proportion of fuzzy successes that are genuinely semantically correct as judged by whether the prediction conveys the same attribute value as the intended canonical value. Specifically test the '16.9 oz' vs '16.9 fl oz' case and similar unit-mismatch cases. If the corrected proportion falls significantly below 0.95, the 23.2% noise-rate estimate is not a valid measure of ground-truth noise and the paper's central claim must be revised. Also run a sensitivity analysis varying τ (e.g., 80, 90, 95) to see how η changes; a sharp drop would confirm threshold artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that exact-match evaluation systematically understates LLM capability rests on the noise-audit estimate η = P(semantically correct | exact-match failure) = 0.232. This estimate is computed using the fuzzy protocol ϕ(80)_FM, which labels a prediction 'correct' whenever sim(norm(ŷ), norm(y)) ≥ 80, with sim defined as the maximum partial ratio over contiguous substrings (Eq. 3). This operationalization is not a faithful proxy for semantic correctness. The partial-ratio definition in Eq. 3 takes the maximum similarity over any substring of length |ŷ| in y. Consequently, a prediction that is merely a substring of the ground truth — e.g., 'Black' when the ground truth is 'Black and Blue' — receives sim=100 and is counted as semantically correct, even though the prediction is incomplete and the product has two colors. Similarly, a prediction that shares a short common substring can pass the threshold. The paper's own Table V includes '16.9 oz' vs '16.9 fl oz' with sim=83, which is at best a unit mismatch (weight vs volume) and demonstrates the permissiveness of the threshold. The manual audit of only 50 'representative' cases is far too small to validate the other 687 fuzzy successes; if even a modest fraction of those are false positives, the true semantic-correctness rate could be substantially lower than 0.232. The paper acknowledges this in the limitations (Sec. VI.E: 'fuzzy matching may admit false positives') but then proceeds to treat η as a 'ground truth noise rate' and uses it to infer that exact-match evaluation understates model capability. This inference is load-bearing: if η is inflated, the claim that benchmark labels are noisy to a degree that dominates model/prompt effects is unsupported. The variance-dominance finding (Sec. V.F) is a separate empirical observation about F1 ranges, but the 'noise rate' interpretation is what gives it the normative weight in the conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a controlled experiment on the MAVE benchmark in which two LLMs (GPT-4o-mini and Gemini 2.5 Flash) and four prompting strategies are evaluated on 6,400 attribute-level predictions under both exact-string and fuzzy-string matching. The paper claims that the choice of evaluation protocol (exact vs. fuzzy) produces F1 differences approximately 23 times larger than model choice and 5 times larger than prompting-strategy choice, and that the MAVE benchmark has a 23.2% 'ground truth noise rate,' defined as η=P(semantically correct | exact-match failure). The authors recommend adopting fuzzy matching as a standard evaluation protocol and argue that evaluation methodology and data quality dominate model and prompt selection for cost-efficient extraction pipelines.","tokens_in":11561,"tokens_out":8388,"duration_ms":74204,"significance":"The central empirical question—whether evaluation methodology can dominate model and prompt choice in LLM-based extraction—is practically important. The direct finding that switching from exact to fuzzy matching changes F1 by 0.109–0.121 while model choice changes it by 0.005 and prompting strategy by up to 0.024 is clearly demonstrated and supported by permutation tests and bootstrap CIs. The study is transparent: code, prompts, and processed data are made public, and the authors explicitly discuss several limitations. However, the two headline contributions are unevenly supported. The protocol-dominance result is directly backed by the data; the 'ground truth noise rate' claim is not, because η is defined as a reclassification rate under the very fuzzy protocol that is being advocated, and the manual validation is too small and permissive to establish semantic correctness. The variance-decomposition language also overstates what is actually a set of marginal ranges.","major_comments":[{"comment":"Eq. (9) defines η=P(S|E), where S is success under the fuzzy protocol. This is a reclassification rate, not a ground-truth noise rate. The sentence in V-E, 'The conditional probability that an exact-match failure is in fact semantically correct is therefore η=0.232,' is an invalid inference unless the fuzzy protocol is a validated proxy for semantic correctness. The only validation is manual inspection of 50 cases, which is small and, as Table V shows, permissive: e.g., 'Purple' vs. 'Purple and silver' and 'Mesh' vs. 'Mesh and synthetic' both score 100, although the model output captures only part of a multi-value attribute. The abstract and conclusion repeatedly refer to a 23.2% ground truth noise rate; this should be either replaced by a properly estimated semantic-correctness rate from a larger, independent, random sample with inter-annotator reliability, or restricted to the claim th","section":"III-D and V-E"},{"comment":"Eq. (10) and Table VI do not constitute a variance decomposition. The table reports marginal ranges of conditional means, and the factors are measured under different subsets: model, strategy, attribute, and category are evaluated only under fuzzy matching, while protocol spans both exact and fuzzy. The ratio in Eq. (20) is therefore a ratio of ranges, not of variances. The abstract's statement that 'evaluation methodology produces variance approximately 23 times larger than model choice' is not supported by a variance model. Either fit a mixed-effects or ANOVA model that attributes components of total variance to each factor, or rephrase all such claims in terms of 'range of mean F1 differences.' This is central because the 23x headline is based on this informal measure.","section":"III-E and V-F"},{"comment":"The claim of practical equivalence between the two models rests on overlapping 95% confidence intervals. Overlap of individual CIs is not a valid test of equivalence; it is consistent with a true difference as well as with noise. Given that the paired permutation test yields p=0.047, the evidence is borderline. Report a confidence interval for the mean within-pair difference and, if equivalence is the intended claim, use a pre-specified equivalence test (e.g., TOST). This does not affect the main protocol-dominance finding but should be corrected.","section":"V-B, Test B"},{"comment":"The scope of the empirical design is limited to 200 products from 5 of MAVE's 1,257 categories, four prompting strategies, two cost-efficient models, and a single fuzzy threshold τ=80. The limitations section acknowledges this, but the abstract and conclusion state the dominance result unconditionally. The strong generalization ('evaluation methodology and data quality dominate the impact of model selection and prompt engineering') should be qualified with the sampled categories, models, and threshold, or supported with evidence of robustness across a wider range of conditions.","section":"IV-B and VI-E"}],"minor_comments":[{"comment":"The two versions of the abstract report different ratios. The full-text abstract says 'five times larger than model choice' and 'four times larger than prompting strategy,' while Section V-F and Table VI report 22.8x and 5.1x. Align the abstract with the results.","section":"Abstract"},{"comment":"The formalization of the partial ratio as a maximum over substrings of length |s1| is a simplification of rapidfuzz's partial_ratio algorithm. Specify the exact version of rapidfuzz used and confirm that Eq. (3) matches the library's implementation, or adjust the formula.","section":"III-B, Eq. (3)"},{"comment":"In the GPT-4o-mini fuzzy column, FEW_SHOT and SCHEMA_GUIDED both achieve F1=0.618, but only FEW_SHOT is bolded. Clarify the bolding rule or note ties.","section":"Table I"},{"comment":"The phrase 'performance degrades monotonically with attribute implicitness' (Fig. 2) is subjective. Define 'implicitness' operationally or replace with a direct statement of the observed ordering.","section":"V-C"},{"comment":"Cohen's kappa between exact and fuzzy protocols is described as indicating 'substantial agreement.' This is standard, but note that kappa between a metric and its relaxed version does not independently validate the relaxed metric; both protocols necessarily agree on exact successes, and the value is inflated by the high number of exact failures that are also fuzzy failures.","section":"V-E"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core: the protocol-dominance effect is real in this sample and the statistical testing is careful. The main problem is the overinterpretation of the fuzzy-reclassification rate as 'ground truth noise,' and the misuse of the term 'variance decomposition.' These are fixable. I would encourage the authors to reframe the noise contribution as a reclassification analysis, conduct a larger and more rigorous manual audit if they want to retain the noise claim, and either implement a proper variance-components model or soften the 'variance' language. No concerns about data integrity or citation behavior were evident."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper has one solid empirical result — for two cost-efficient LLMs on a 200-product MAVE sample, switching from exact to fuzzy matching moves F1 by ~0.12, while model choice moves it by ~0.005 and prompting strategy by ~0.024. That is a useful measurement for anyone deploying LLM extraction pipelines. The rest is a mix of good framing and overreach.\n\nWhat is actually new: the quantified spread on MAVE for this model tier, with 6,400 attribute-level predictions and paired permutation tests, plus public code and data. Table I and the per-attribute/per-category results are clear. The authors also deserve credit for listing limitations honestly in Sec. VI.E.\n\nThe soft spot is the noise audit. The reader's concern lands. Eq. (9) defines η as the proportion of exact-match failures that the fuzzy protocol relabels as correct. Calling that \"ground truth noise\" requires fuzzy matching to be a faithful proxy for semantic correctness, and the partial-ratio measure is not. Substring matches get sim=100, so \"Black\" against \"Black and Blue\" counts as correct. The paper's own example \"16.9 oz\" vs \"16.9 fl oz\" at sim=83 is a unit mismatch. Hand-checking 50 of 737 cases is too small to validate the rest. The limitation section says fuzzy matching may admit false positives — then the conclusion treats η as proof of understatement. That is a circular step, and the normative claim about \"data quality\" leans on it.\n\nTwo smaller issues: the abstract says \"five times\" in one place and \"23 times\" in another — internal inconsistency, easy fix. The \"variance decomposition\" is just spans, not a real decomposition; drop the word \"formal.\" The Gemini few-shot vs schema comparison (p=0.003) is post-hoc best-vs-worst without multiple-comparison control; minor.\n\nThe good news is the central direction holds without the noise-rate interpretation. The inter-protocol F1 gap of 0.121 is a measured fact, and it is far larger than model or prompt effects in this setup. That alone supports the conclusion that exact-match evaluation understates capability. The paper does not need the 23.2% headliner.\n\nWho this is for: practitioners benchmarking extraction models, and anyone writing evaluation methodology for generative IE. I would bring it to a reading group and would cite the inter-protocol finding, though not the noise rate.\n\nRecommendation: send it to peer review. It is a real empirical contribution with fixable framing. A revision that reframes η as \"the rate at which fuzzy matching disagrees with exact matching,\" adds threshold sensitivity analysis, and expands the semantic audit would make the claims match the evidence.","headline":"The evaluation-methodology finding is real and useful — but the 23.2% 'ground truth noise rate' is a fuzzy-match artifact, not an audit.","tokens_in":12129,"tokens_out":2348,"would_cite":true,"duration_ms":24853,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Evaluation methodology, not model choice, drives reported F1 variance in LLM attribute extraction.","keywords":["product attribute extraction","large language models","evaluation methodology","fuzzy matching","exact match","ground truth noise","variance decomposition","prompting strategies"],"falsifier":"Have independent annotators, blind to protocol, classify all 737 exact-match-failure/fuzzy-match-success predictions as semantically correct or incorrect. If far fewer than 23% are judged correct, the noise-rate claim fails. Likewise, recomputing η across thresholds τ = 60 to 95 would reveal whether the result is driven by the specific threshold choice.","tokens_in":11055,"feed_emoji":"🔍","tokens_out":3397,"duration_ms":32330,"temperature":0.7,"pith_summary":"This paper argues that how you score a model's extractions changes reported performance far more than which model or prompt you use. In a controlled study of two production-grade LLMs and four prompting strategies on the MAVE product-attribute benchmark, switching from exact string matching to fuzzy matching raises F1 by roughly 12 points — about 23 times the effect of model choice and 5 times the effect of prompt choice. The authors also report a 23.2% ground-truth noise rate: nearly a quarter of predictions judged wrong by exact match are actually semantically correct, just differently formatted. If correct, these results mean reported LLM capability is systematically understated, and practitioners should fix evaluation before optimizing models or prompts.","feed_headline":"Fuzzy scoring moves F1 23x more than swapping models","feed_subtitle":"A 6,400-prediction study shows exact-match grading hides a 23% ground-truth noise rate.","key_machinery":"The pair of scoring protocols: exact match after normalization, and fuzzy match using a Levenshtein-based partial-ratio similarity with threshold τ = 80. The noise audit formalizes η = P(fuzzy success | exact failure), and the variance decomposition measures the span of mean F1 within each factor (protocol, model, strategy, attribute, category). These two tools — the conditional noise rate and the span-based decomposition — carry the argument that methodology dominates controllable factors.","core_discovery":"The central discovery is a quantitative dominance result: evaluation methodology, not model or prompt selection, drives the variance in reported F1 for cost-efficient LLM attribute extraction. Formalized as the conditional probability η = P(semantically correct | exact-match failure) = 0.232, with substantial inter-protocol agreement κ = 0.769, the paper shows that exact-match failures are frequently correct predictions whose values differ in completeness or format. The inter-protocol F1 gap (ΔF1 = 0.121, p < 0.0001) is attributed to genuine semantic correctness rather than scoring looseness, and the paper concludes that fuzzy matching should be a standard evaluation protocol for LLM-based e","pith_inferences":["If the noise-rate finding generalizes, published leaderboards for generative extraction tasks may be re-ranked under semantic evaluation, and exact-match F1 gaps between papers may reflect scoring protocol rather than method quality.","The τ = 80 threshold is a free parameter; future work varying τ from 60 to 95 could show that the 23.2% noise rate is threshold-sensitive, which would change the practical recommendation.","The same audit methodology could be applied to other generative NLP tasks with single canonical reference strings — question answering, key-value extraction, data cleaning — to measure benchmark noise there.","Fuzzy matching can also admit false positives; the paper's manual check covered 50 of 737 success cases, so an extended blind audit would tighten the estimate of true semantic correctness."],"forward_implications":["Practitioners who evaluate with exact matching will systematically under-report the capability of cost-efficient LLMs and may discard a model that is actually adequate.","Published model comparisons are largely incomparable unless they use the same scoring protocol, since exact-vs-fuzzy differences can swamp real model differences.","Prompt engineering yields near-zero return for some model families (GPT-4o-mini, p = 1.0) while few-shot prompting helps others (Gemini 2.5 Flash, +2.4 points, p = 0.003).","Benchmark ground truth built from single canonical values will mislabel valid alternative answers; the paper recommends fuzzy or semantic evaluation as the default.","Attribute and category difficulty (brand 1.0 vs material ~0.4; Sports 0.74 vs Electronics 0.46) dominate model and strategy effects, so evaluation design should focus on hard attributes."],"fun_headline_variants":["Evaluation method beats model choice by 23x in F1 variance","Exact-match grading hides 23% noise in LLM attribute extraction","Fuzzy scoring shifts F1 23x more than picking a different LLM","For product extraction, evaluation protocol outweighs model choice","Why exact match misleads: 23% label noise, 23x F1 gap"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a partial-ratio fuzzy similarity score of 80 or higher is a faithful proxy for semantic correctness; if fuzzy matching admits many wrong values above threshold, the 23.2% noise rate and the variance-dominance conclusion rest on the measuring instrument rather than on true benchmark noise.","fun_headline_variants_meta":{"raw":{"variants":["Evaluation method beats model choice by 23x in F1 variance","Exact-match grading hides 23% noise in LLM attribute extraction","Fuzzy scoring shifts F1 23x more than picking a different LLM","For product extraction, evaluation protocol outweighs model choice","Why exact match misleads: 23% label noise, 23x F1 gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1152,"prompt_tokens":761,"completion_tokens":391,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":293}},"tokens_in":505,"tokens_out":391,"duration_ms":3760,"temperature":1.0,"reasoning_tokens":293,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:02:17.687535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators, blind to protocol, classify all 737 exact-match-failure/fuzzy-match-success predictions as semantically correct or incorrect. If far fewer than 23% are judged correct, the noise-rate claim fails. Likewise, recomputing η across thresholds τ = 60 to 95 would reveal whether the result is driven by the specific threshold choice.","supporting_citations":[],"review_version":1}