{"id":"0fed4ed2-cfed-4ce2-b13c-9b1668c88ca1","arxiv_id":"2412.13378","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark built with executable phrase-level edits shows that most LLMs detect and explain factual inconsistencies in summaries only weakly, with the best model scoring 0.49 on the joint task.","lead":"This paper introduces SummExecEdit, a benchmark of 4,241 document-summary pairs in which small, hard-to-spot factual errors are inserted into summaries by replacing a specific phrase with an inconsistent one. It reports that even the best tested large language model, Claude3-Opus, scores only 0.49 on the combined task of detecting the error and explaining it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The final benchmark's 2,121 'inconsistent' summaries are never human-validated for actual inconsistency or explanation correctness; the POC annotation suggests only 44-49% of executable edits pass all quality filters, so the reported 0.49 JS is not a reliable measure.","rationale":"All downstream numbers in the paper—detection accuracy, explanation score, joint score, and the qualitative claim that 'the task is still challenging'—are computed against the benchmark's labels and reference explanations. If those labels are noisy, every headline number is suspect. The authors themselves validate only a small POC set (Section 3, Table 1) and a small explanation subset (Section 5.2); the final dataset has no human validation of inconsistency or explanation correctness. The GPT4-Turbo trivial-edit filter (F.5) is a proxy, not a validator, and its accuracy is not reported. The POC's cascade percentages show that a large fraction of generated edits do not survive human quality checks, making it likely that the unfiltered final set contains a meaningful number of invalid edits. A concrete manual audit of a random sample from the final benchmark would directly measure this fraction. If the fraction is low, the concern is mitigated; if high, the scores need to be recomputed. This is more load-bearing than, say, the overlap between generating and evaluated models, because even a perfect model evaluation protocol cannot rescue a benchmark with invalid ground truth. The paper's own Limitations section does not acknowledge this validation gap, which further supports the need for an independent audit before the difficulty numbers are taken as definitive.","tokens_in":12358,"tokens_out":8879,"duration_ms":72955,"concrete_test":"Randomly sample 300 of the 2,121 'inconsistent' summaries (stratified by domain and generating model) and 300 of the 2,120 'consistent' summaries, and have two independent annotators mark (a) whether the edited summary is factually inconsistent with the document, and (b) whether the reference explanation correctly identifies the specific inconsistency. Compute the estimated false-inconsistency rate and explanation-error rate with the two-annotator agreement. If the false-inconsistency rate is >10% or the explanation-error rate >20%, recompute DA, DS, ES, and JS on the clean subset; if JS shifts by more than 0.05, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (Claude3-Opus JS=0.49) depends entirely on the ground-truth quality of the 2,121 factually inconsistent summaries. The full pipeline does not include any manual verification of the final edits: after generating six edits per (document, summary) pair with GPT4-Turbo and Claude3-Opus, the only filter is GPT4-Turbo's coarse categorization of an edit as DATE_CHANGE, NUMBER_CHANGE, ANTONYM_CHANGE, or OTHER (Prompt F.5), and the first three categories are removed. This filter does not test whether the edited summary is actually inconsistent with the document, whether the edit is complex/challenging, or whether the reference explanation is correct. The POC manual annotation (Table 1) indicates that, even in the best executable condition, only 48.55% of Claude3-Opus and 44.44% of GPT4-Turbo generated edits meet all four criteria (controlled, inconsistent, complex, good explanation). If the final benchmark retains unselected generated edits, a substantial fraction could be consistent or trivial. Additional evidence: only 40 reference explanations were manually checked (score 0.95), and the POC inter-annotator kappa for explanation quality is only 0.49 (Table 5). If even 10-20% of the 2,121 'inconsistent' summaries are actually consistent, the detection and joint scores are no longer measuring what the paper claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SummExecEdit, a benchmark for detecting and explaining factual inconsistencies in summarization. The authors propose executable edits, in which an LLM isolates a substring of an original summary, replaces it with an inconsistent alternative, and provides a reference explanation. They compare executable vs. non-executable editing in a proof-of-concept with manual annotation, filter generated edits by removing trivial date, number, and antonym changes with GPT4-Turbo, and evaluate 20+ LLMs under two prompts (detect-and-explain, and explain-given-detection). The best-reported joint score is Claude3-Opus at 0.49, with 0.67 detection and 0.733 explanation, and the paper provides an error taxonomy for incorrect explanations.","tokens_in":12615,"tokens_out":9734,"duration_ms":84461,"significance":"If the benchmark ground truth is reliable, SummExecEdit is a useful resource: it provides controlled phrase-level edits with reference explanations, releases the data, includes a proof-of-concept human annotation and explanation-judge validation, and offers a clear error taxonomy. The finding that top LLMs achieve only moderate joint scores would be a meaningful challenge result for the community. However, the central evaluation claim rests on the unvalidated quality of the final benchmark's inconsistency labels and reference explanations; the paper's own proof-of-concept numbers and low explanation inter-annotator agreement indicate that this risk is substantial. The contribution is real but requires additional validation before the headline numbers can be taken at face value.","major_comments":[{"comment":"The final 2,121 'inconsistent' summaries are filtered only by GPT4-Turbo's category assignment (DATE_CHANGE, NUMBER_CHANGE, ANTONYM_CHANGE, OTHER), which does not verify that the edited summary is actually inconsistent with the document, that the edit is non-trivial, or that the paired reference explanation is correct. The proof-of-concept manual annotation in Table 1 shows that under the best executable condition only 48.55% (Claude3-Opus) and 44.44% (GPT4-Turbo) of generated edits receive a positive judgment on explanation quality, with inconsistency rates of 84.78% and 81.25% and complexity rates of 49.28% and 45.83%. If a substantial fraction of the 2,121 samples are actually consistent, trivially edited, or paired with incorrect reference explanations, then the reported DS, ES, and JS values in Table 4 do not measure the paper's stated construct. The authors should report a human validation study on a random sample of the final benchmark with precision of the inconsistency labels and explanation correctness, and re-estimate the scores on the validated subset.","section":"Section 4, Prompt F.5"},{"comment":"The ground-truth reference explanations are not sufficiently validated. Only 40 reference explanations were manually checked, and the inter-annotator agreement for explanation-quality is Cohen's Kappa 0.49, which is low. Because ES and JS are computed against these reference explanations using EvalV4 with GPT4o as judge, unreliable references could substantially change the model ranking and the claim that the best JS is only 0.49. The paper should provide a larger manual validation of reference explanations and report the judge's agreement with human judgments on the final benchmark, not only on the 887-example explanation-evaluation subset.","section":"Section 5.2, Table 5"},{"comment":"The benchmark's edits and reference explanations are generated by GPT4-Turbo and Claude3-Opus, and Claude3-Opus is the top-scoring model on the joint task. The paper does not include a cross-generation control. A model may be advantaged when evaluating edits and reference explanations produced by the same generation pipeline or family. To make the ranking claim robust, the authors should partition results by the generator of each sample (e.g., GPT4-Turbo-generated vs. Claude3-Opus-generated edits) and, ideally, evaluate a held-out model family not used in generation. This would also address whether the reported 0.49 JS is an artifact of generator identity.","section":"Sections 3-4, Table 4"},{"comment":"The description of Table 1 is internally ambiguous. The text says 'each subsequent column filters out the edits deemed inappropriate by either of the annotators in the previous column,' while the caption says each column 'does not consider the edits filtered out in its previous columns.' These readings lead to very different cumulative pass rates (e.g., under an independent-column reading, the product of the four rates is roughly 14% for GPT4-Turbo Exec and 19% for Claude3-Opus Exec). Since the superiority of executable edits is a central motivating claim, the authors should clarify the exact computation of each column and report the cumulative pass rate after all four filters.","section":"Section 3, Table 1"}],"minor_comments":[{"comment":"The conclusion states 'Claude3.7-Sonnet achieves an accuracy of 73%' but Table 4 reports 'claude3.5-sonnet' with DA 0.737; please correct the model name or number.","section":"Section 6 / Table 4"},{"comment":"The text first says 65 samples were misclassified by all LLMs and then says 'Among these, 60 samples belonged to the category of inconsistent summaries'; earlier it mentions 60 inconsistent summaries. Please reconcile the 60/65 counts.","section":"Section 5.1"},{"comment":"The instruction for explanations reads 'The document says X but the document says Y,' which should presumably be 'but the summary says Y.'","section":"Prompt F.1"},{"comment":"The guidelines say 'You can generate only one error per edited you generate' and 'For each of the three edits' while requesting six edits; please standardize the wording.","section":"Prompt F.2"},{"comment":"The number 350 manually annotated incorrect explanations should be reconciled with the 887 explanations in Table 6 and the 'around 1200' explanations mentioned in Section 5.2.","section":"Section 5.4 / Table 6"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the gap between the proof-of-concept validation and the final benchmark construction; a human validation study of the final benchmark is necessary. I see no inappropriate citation practices or novelty disclosure concerns; the scope is appropriate for a benchmark paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it's a serious benchmark contribution: the executable-editing protocol is a real improvement over the direct-editing approach in SummEdits, and the authors show with manually annotated data that it yields more controlled, more complex inconsistencies. Second, the headline result—Claude3-Opus gets a joint score of only 0.49—is not yet trustworthy as a measure of model capability, because the ground truth for the final 2,121 inconsistent summaries was never human-validated in full. The stress-test note is on target here: the POC annotation shows that only about 44–49% of generated edits pass all four quality criteria, and the final pipeline relies on a GPT4-Turbo filter that only removes date, number, and antonym changes. That filter does not catch trivial edits of other kinds, nor does it check whether the edited summary is actually inconsistent with the document. So a fair reading is that the benchmark likely contains a nontrivial fraction of summaries that are either consistent or trivially inconsistent. That makes the reported detection and joint scores optimistic, not pessimistic.\n\nThe paper does several things well. It releases the data and prompts, which is necessary for this kind of work. It evaluates 20+ models with two prompt settings and shows a clear gap between open-source and API models. The error taxonomy for explanations—misattribution, additional irrelevant explanation, completeness focus, vague explanation—is useful and actionable. The comparison between executable and non-executable editing is a genuine empirical contribution; the manual annotation for that comparison is transparent and the IAA for the \"inconsistent\" question (kappa 0.76) is respectable.\n\nThe soft spots are real but not fatal. The explanation-quality IAA is low (kappa 0.49), and the reference explanations were only checked on 40 samples. The overlap between edit-generating models (GPT4-Turbo, Claude3-Opus) and evaluated models means those two models' scores are partly circular, though the effect is hard to quantify. There are no confidence intervals for any of the model comparisons, which matters when differences are small. The authors acknowledge several limitations in their own limitation section, but they do not flag the validation gap for the final benchmark, which is the most important one.\n\nWho should read this? Anyone building or using factual-consistency benchmarks, and anyone evaluating LLM hallucination detection. It will likely become a reference point for executable-edit-based benchmark construction. It deserves a serious referee: the idea is sound, the release is complete, and the flaws are the kind that can be fixed with additional validation, not fundamental.\n\nRecommendation: send to peer review, but require a human-validated sample of the final benchmark (say 300–500 examples) with per-category quality scores, plus confidence intervals for the main numbers. That would make the difficulty claims defensible.","headline":"A genuinely useful benchmark with a real design idea, but the headline difficulty numbers rest on ground truth that was only partially validated.","tokens_in":13190,"tokens_out":1671,"would_cite":true,"duration_ms":18912,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that executable edits, which isolate a single substring replacement and its explanation, produce a harder and more interpretable factual-consistency benchmark for summarization, and that the best model scores only 0.49 on…","keywords":["factual consistency","summarization","hallucination detection","executable edits","benchmark","LLM evaluation","explanation quality"],"falsifier":"Randomly sample 200 of the 2,121 'inconsistent' summaries from SummExecEdit and have independent annotators (not the authors) judge whether each one is truly inconsistent with its document. If the rate of true inconsistency is well below 100% — for instance, below 80% — then the reported detection and joint scores are partly measuring label noise rather than model ability.","tokens_in":12127,"feed_emoji":"📋","tokens_out":8767,"duration_ms":66409,"temperature":0.7,"pith_summary":"The paper introduces SummExecEdit, a benchmark for factual consistency in summarization built from 'executable edits' — instructions that specify exactly which substring of a summary to replace and why the replacement creates an inconsistency. The authors argue that this editing style produces more challenging and more interpretable test items than earlier benchmarks, which often introduced multiple broad, easy-to-spot changes. On the 4,241-sample benchmark, the best model (Claude3-Opus) detects inconsistencies correctly on 0.67 of inconsistent summaries and explains them on 0.73, but the combined joint score is only 0.49, and more than half of the 20+ LLMs tested fail on over 30% of the samples. The paper also categorizes the explanation errors produced by models, finding that 45.4% blame a completely unrelated part of the summary.","feed_headline":"Best LLM scores 0.49 on new summary-consistency test","feed_subtitle":"Modern LLMs detect and explain fewer than half of the benchmark's subtle summary inconsistencies.","key_machinery":"The central object is the executable edit: an editing operation that names the exact substring of the original summary to be replaced, the replacement substring, and a short explanation of the resulting inconsistency. By making the edit a program that can be applied directly, the pipeline keeps the original summary mostly intact, forces the model to locate a single granular change, and produces a structured reference explanation for each edit. This is what makes the benchmark both more controlled (fewer broad, sweeping changes) and more interpretable (each sample carries an explanation of the intended error). The benchmark construction also relies on a trivial-edit filter (a GPT4-Turbo classifier that removes date, number, and antonym changes) and on reference explanations generated at edit time.","core_discovery":"The central claim is that executable edits — having a language model output a specific span to replace, the replacement text, and an explanation, instead of rewriting the whole summary — yield a factual-consistency benchmark that is both harder and more interpretable than prior direct-edit benchmarks. The paper demonstrates this in a small proof-of-concept with human annotation (executable edits produce 18–25% more complex, controlled edits with better explanations), then scales up to create SummExecEdit from the SummEdits corpus, filtering out trivial date, number, and antonym changes with GPT4-Turbo. On this benchmark, the strongest tested model, Claude3-Opus, achieves a joint detection-and-explanation score of 0.49 (detection 0.67, explanation 0.73 on inconsistent summaries), and most of the 20+ evaluated LLMs perform substantially worse, with over half struggling on more than 30% of the samples. The paper further shows that explanation evaluation works best when a reference explanation is provided, and that model explanations fail in four recurring ways, the most common (45.4%) being misattribution of the error to an unrelated part of the summary.","pith_inferences":["The benchmark's reliance on a GPT4-Turbo trivial-edit filter without reported accuracy leaves open the possibility that some easy edits survive, which would inflate the challenge gap; auditing the filtered-out and kept edits would settle this.","The low inter-annotator agreement on explanation quality (Cohen's Kappa 0.49) suggests the reference explanations themselves are noisy; if so, the reported explanation and joint scores may underestimate model capability on cleaner labels.","Executable editing could transfer to other generation tasks where inconsistency is defined against a source, such as data-to-text generation, dialogue state tracking, or fact-checking of model outputs, since the same substring-replacement-plus-explanation structure applies.","The 45.4% misattribution rate hints that models often locate errors in the wrong span even when they flag the right summary, so explanation quality may be a better probe of reasoning than binary detection."],"forward_implications":["If executable edits produce harder, more interpretable inconsistencies, benchmark creators can generate challenging factuality tests at scale without human editing, at least when ground-truth document-summary pairs exist.","Current LLMs are far from reliable at the joint task of detecting and explaining inconsistencies; the best joint score of 0.49 suggests deployment of summary-consistency checking in real systems should not rely on a single model.","The finding that reference explanations improve LLM-as-judge evaluation (EvalV4) implies that benchmarks for hallucination explanation should include reference explanations to get reliable automatic scoring.","The four explanation error types (misattribution, extra irrelevant explanation, completeness focus, vague explanation) give concrete targets for improving model reasoning over facts.","The large gap between detection and explanation scores for some models (e.g., Llama3.1-8b detects well but explains poorly) shows these abilities are dissociable and should be evaluated separately."],"supporting_citations":[{"why":"Supplies the seed (document, summary) pairs and the consistent summaries that make up half of SummExecEdit; the benchmark is built on this corpus.","marker":"(Laban et al., 2023)"},{"why":"Introduces executable editing, the central mechanism the paper adapts to generate controlled, granular inconsistency edits.","marker":"(Laban et al., 2024)"},{"why":"AlignScore serves as a non-LLM detection baseline scoring 57.4%, used to show SummExecEdit is harder than prior benchmarks.","marker":"(Zha et al., 2023)"},{"why":"MiniCheck is the other non-LLM baseline at 60.0%, providing a compute-light comparison point for detection.","marker":"(Tang et al., 2024a)"},{"why":"LLM-as-judge methodology used to evaluate the quality of model-generated explanations with different prompt variants.","marker":"(Zheng et al., 2023)"}],"fun_headline_variants":["Executable edits make summary-consistency tests harder and clearer","Claude3-Opus tops new summary benchmark with 0.49 joint","Most LLMs fail 30% of SummExecEdit, best joint score 0.49","New benchmark: executable edits expose summary consistency weaknesses","SummExecEdit: a tougher, more interpretable fact-consistency test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that each LLM-generated replacement genuinely makes the edited summary inconsistent with its document and that the accompanying reference explanation is correct, but only a few hundred of the thousands of edits were manually verified, and the two annotators disagreed substantially on explanation quality.","fun_headline_variants_meta":{"raw":{"variants":["Executable edits make summary-consistency tests harder and clearer","Claude3-Opus tops new summary benchmark with 0.49 joint","Most LLMs fail 30% of SummExecEdit, best joint score 0.49","New benchmark: executable edits expose summary consistency weaknesses","SummExecEdit: a tougher, more interpretable fact-consistency test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001686,"raw_usage":{"total_tokens":6676,"prompt_tokens":936,"completion_tokens":5740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":5642}},"tokens_in":552,"tokens_out":5740,"duration_ms":37370,"temperature":1.0,"reasoning_tokens":5642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:11:07.780947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample 200 of the 2,121 'inconsistent' summaries from SummExecEdit and have independent annotators (not the authors) judge whether each one is truly inconsistent with its document. If the rate of true inconsistency is well below 100% — for instance, below 80% — then the reported detection and joint scores are partly measuring label noise rather than model ability.","supporting_citations":[],"review_version":1}