SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits

Chien-Sheng Wu; Onkar Thorat; Philippe Laban

REVIEW 4 major objections 5 minor 22 references

SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that executable edits, which isolate a single substring replacement and its explanation, produce a harder and more interpretable factual-consistency benchmark for summarization, and that the best model scores only 0.49 on…

desk verdict A genuinely useful benchmark with a real design idea, but the headline difficulty numbers rest on ground truth that was only partially validated. read the letter →

arxiv 2412.13378 v2 pith:MVL6Q7EF submitted 2024-12-17 cs.CL

Onkar Thorat , Philippe Laban , Chien-Sheng Wu This is my paper · ORCID

classification cs.CL

keywords factualconsistencysummarizationhallucinationdetectionexecutableeditsbenchmarkLLMevaluationexplanationquality

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SummExecEdit, a benchmark for factual consistency in summarization built from 'executable edits' — instructions that specify exactly which substring of a summary to replace and why the replacement creates an inconsistency. The authors argue that this editing style produces more challenging and more interpretable test items than earlier benchmarks, which often introduced multiple broad, easy-to-spot changes. On the 4,241-sample benchmark, the best model (Claude3-Opus) detects inconsistencies correctly on 0.67 of inconsistent summaries and explains them on 0.73, but the combined joint score is only 0.49, and more than half of the 20+ LLMs tested fail on over 30% of the samples. The paper also categorizes the explanation errors produced by models, finding that 45.4% blame a completely unrelated part of the summary.

What carries the argument

The central object is the executable edit: an editing operation that names the exact substring of the original summary to be replaced, the replacement substring, and a short explanation of the resulting inconsistency. By making the edit a program that can be applied directly, the pipeline keeps the original summary mostly intact, forces the model to locate a single granular change, and produces a structured reference explanation for each edit. This is what makes the benchmark both more controlled (fewer broad, sweeping changes) and more interpretable (each sample carries an explanation of the intended error). The benchmark construction also relies on a trivial-edit filter (a GPT4-Turbo classifier that removes date, number, and antonym changes) and on reference explanations generated at edit time.

What would settle it

Randomly sample 200 of the 2,121 'inconsistent' summaries from SummExecEdit and have independent annotators (not the authors) judge whether each one is truly inconsistent with its document. If the rate of true inconsistency is well below 100% — for instance, below 80% — then the reported detection and joint scores are partly measuring label noise rather than model ability.

Watch

Extended reading notes

Core claim

The central claim is that executable edits — having a language model output a specific span to replace, the replacement text, and an explanation, instead of rewriting the whole summary — yield a factual-consistency benchmark that is both harder and more interpretable than prior direct-edit benchmarks. The paper demonstrates this in a small proof-of-concept with human annotation (executable edits produce 18–25% more complex, controlled edits with better explanations), then scales up to create SummExecEdit from the SummEdits corpus, filtering out trivial date, number, and antonym changes with GPT4-Turbo. On this benchmark, the strongest tested model, Claude3-Opus, achieves a joint detection-and-explanation score of 0.49 (detection 0.67, explanation 0.73 on inconsistent summaries), and most of the 20+ evaluated LLMs perform substantially worse, with over half struggling on more than 30% of the samples. The paper further shows that explanation evaluation works best when a reference explanation is provided, and that model explanations fail in four recurring ways, the most common (45.4%) being misattribution of the error to an unrelated part of the summary.

Load-bearing premise

The benchmark assumes that each LLM-generated replacement genuinely makes the edited summary inconsistent with its document and that the accompanying reference explanation is correct, but only a few hundred of the thousands of edits were manually verified, and the two annotators disagreed substantially on explanation quality.

Editorial extensions

If this is right

If executable edits produce harder, more interpretable inconsistencies, benchmark creators can generate challenging factuality tests at scale without human editing, at least when ground-truth document-summary pairs exist.
Current LLMs are far from reliable at the joint task of detecting and explaining inconsistencies; the best joint score of 0.49 suggests deployment of summary-consistency checking in real systems should not rely on a single model.
The finding that reference explanations improve LLM-as-judge evaluation (EvalV4) implies that benchmarks for hallucination explanation should include reference explanations to get reliable automatic scoring.
The four explanation error types (misattribution, extra irrelevant explanation, completeness focus, vague explanation) give concrete targets for improving model reasoning over facts.
The large gap between detection and explanation scores for some models (e.g., Llama3.1-8b detects well but explains poorly) shows these abilities are dissociable and should be evaluated separately.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The benchmark's reliance on a GPT4-Turbo trivial-edit filter without reported accuracy leaves open the possibility that some easy edits survive, which would inflate the challenge gap; auditing the filtered-out and kept edits would settle this.
The low inter-annotator agreement on explanation quality (Cohen's Kappa 0.49) suggests the reference explanations themselves are noisy; if so, the reported explanation and joint scores may underestimate model capability on cleaner labels.
Executable editing could transfer to other generation tasks where inconsistency is defined against a source, such as data-to-text generation, dialogue state tracking, or fact-checking of model outputs, since the same substring-replacement-plus-explanation structure applies.
The 45.4% misattribution rate hints that models often locate errors in the wrong span even when they flag the right summary, so explanation quality may be a better probe of reasoning than binary detection.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, it's a serious benchmark contribution: the executable-editing protocol is a real improvement over the direct-editing approach in SummEdits, and the authors show with manually annotated data that it yields more controlled, more complex inconsistencies. Second, the headline result—Claude3-Opus gets a joint score of only 0.49—is not yet trustworthy as a measure of model capability, because the ground truth for the final 2,121 inconsistent summaries was never human-validated in full. The stress-test note is on target here: the POC annotation shows that only about 44–49% of generated edits pass all four quality criteria, and the final pipeline relies on a GPT4-Turbo filter that only removes date, number, and antonym changes. That filter does not catch trivial edits of other kinds, nor does it check whether the edited summary is actually inconsistent with the document. So a fair reading is that the benchmark likely contains a nontrivial fraction of summaries that are either consistent or trivially inconsistent. That makes the reported detection and joint scores optimistic, not pessimistic.

The paper does several things well. It releases the data and prompts, which is necessary for this kind of work. It evaluates 20+ models with two prompt settings and shows a clear gap between open-source and API models. The error taxonomy for explanations—misattribution, additional irrelevant explanation, completeness focus, vague explanation—is useful and actionable. The comparison between executable and non-executable editing is a genuine empirical contribution; the manual annotation for that comparison is transparent and the IAA for the "inconsistent" question (kappa 0.76) is respectable.

The soft spots are real but not fatal. The explanation-quality IAA is low (kappa 0.49), and the reference explanations were only checked on 40 samples. The overlap between edit-generating models (GPT4-Turbo, Claude3-Opus) and evaluated models means those two models' scores are partly circular, though the effect is hard to quantify. There are no confidence intervals for any of the model comparisons, which matters when differences are small. The authors acknowledge several limitations in their own limitation section, but they do not flag the validation gap for the final benchmark, which is the most important one.

Who should read this? Anyone building or using factual-consistency benchmarks, and anyone evaluating LLM hallucination detection. It will likely become a reference point for executable-edit-based benchmark construction. It deserves a serious referee: the idea is sound, the release is complete, and the flaws are the kind that can be fixed with additional validation, not fundamental.

Recommendation: send to peer review, but require a human-validated sample of the final benchmark (say 300–500 examples) with per-category quality scores, plus confidence intervals for the main numbers. That would make the difficulty claims defensible.

Referee Report

4 major / 5 minor

Summary. The paper introduces SummExecEdit, a benchmark for detecting and explaining factual inconsistencies in summarization. The authors propose executable edits, in which an LLM isolates a substring of an original summary, replaces it with an inconsistent alternative, and provides a reference explanation. They compare executable vs. non-executable editing in a proof-of-concept with manual annotation, filter generated edits by removing trivial date, number, and antonym changes with GPT4-Turbo, and evaluate 20+ LLMs under two prompts (detect-and-explain, and explain-given-detection). The best-reported joint score is Claude3-Opus at 0.49, with 0.67 detection and 0.733 explanation, and the paper provides an error taxonomy for incorrect explanations.

Significance. If the benchmark ground truth is reliable, SummExecEdit is a useful resource: it provides controlled phrase-level edits with reference explanations, releases the data, includes a proof-of-concept human annotation and explanation-judge validation, and offers a clear error taxonomy. The finding that top LLMs achieve only moderate joint scores would be a meaningful challenge result for the community. However, the central evaluation claim rests on the unvalidated quality of the final benchmark's inconsistency labels and reference explanations; the paper's own proof-of-concept numbers and low explanation inter-annotator agreement indicate that this risk is substantial. The contribution is real but requires additional validation before the headline numbers can be taken at face value.

major comments (4)

[Section 4, Prompt F.5] The final 2,121 'inconsistent' summaries are filtered only by GPT4-Turbo's category assignment (DATE_CHANGE, NUMBER_CHANGE, ANTONYM_CHANGE, OTHER), which does not verify that the edited summary is actually inconsistent with the document, that the edit is non-trivial, or that the paired reference explanation is correct. The proof-of-concept manual annotation in Table 1 shows that under the best executable condition only 48.55% (Claude3-Opus) and 44.44% (GPT4-Turbo) of generated edits receive a positive judgment on explanation quality, with inconsistency rates of 84.78% and 81.25% and complexity rates of 49.28% and 45.83%. If a substantial fraction of the 2,121 samples are actually consistent, trivially edited, or paired with incorrect reference explanations, then the reported DS, ES, and JS values in Table 4 do not measure the paper's stated construct. The authors should report a human validation study on a random sample of the final benchmark with precision of the inconsistency labels and explanation correctness, and re-estimate the scores on the validated subset.
[Section 5.2, Table 5] The ground-truth reference explanations are not sufficiently validated. Only 40 reference explanations were manually checked, and the inter-annotator agreement for explanation-quality is Cohen's Kappa 0.49, which is low. Because ES and JS are computed against these reference explanations using EvalV4 with GPT4o as judge, unreliable references could substantially change the model ranking and the claim that the best JS is only 0.49. The paper should provide a larger manual validation of reference explanations and report the judge's agreement with human judgments on the final benchmark, not only on the 887-example explanation-evaluation subset.
[Sections 3-4, Table 4] The benchmark's edits and reference explanations are generated by GPT4-Turbo and Claude3-Opus, and Claude3-Opus is the top-scoring model on the joint task. The paper does not include a cross-generation control. A model may be advantaged when evaluating edits and reference explanations produced by the same generation pipeline or family. To make the ranking claim robust, the authors should partition results by the generator of each sample (e.g., GPT4-Turbo-generated vs. Claude3-Opus-generated edits) and, ideally, evaluate a held-out model family not used in generation. This would also address whether the reported 0.49 JS is an artifact of generator identity.
[Section 3, Table 1] The description of Table 1 is internally ambiguous. The text says 'each subsequent column filters out the edits deemed inappropriate by either of the annotators in the previous column,' while the caption says each column 'does not consider the edits filtered out in its previous columns.' These readings lead to very different cumulative pass rates (e.g., under an independent-column reading, the product of the four rates is roughly 14% for GPT4-Turbo Exec and 19% for Claude3-Opus Exec). Since the superiority of executable edits is a central motivating claim, the authors should clarify the exact computation of each column and report the cumulative pass rate after all four filters.

minor comments (5)

[Section 6 / Table 4] The conclusion states 'Claude3.7-Sonnet achieves an accuracy of 73%' but Table 4 reports 'claude3.5-sonnet' with DA 0.737; please correct the model name or number.
[Section 5.1] The text first says 65 samples were misclassified by all LLMs and then says 'Among these, 60 samples belonged to the category of inconsistent summaries'; earlier it mentions 60 inconsistent summaries. Please reconcile the 60/65 counts.
[Prompt F.1] The instruction for explanations reads 'The document says X but the document says Y,' which should presumably be 'but the summary says Y.'
[Prompt F.2] The guidelines say 'You can generate only one error per edited you generate' and 'For each of the three edits' while requesting six edits; please standardize the wording.
[Section 5.4 / Table 6] The number 350 manually annotated incorrect explanations should be reconciled with the 887 explanations in Table 6 and the 'around 1200' explanations mentioned in Section 5.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical benchmark measurements; the main concerns are ground-truth validity, not derivation circularity.

full rationale

SummExecEdit is constructed by generating executable edits with GPT4-Turbo and Claude3-Opus, filtering trivial edits with GPT4-Turbo, and then measuring LLM detection and explanation performance on the resulting fixed benchmark. No parameter is fitted to a subset and then used to predict the same or a closely related quantity; no load-bearing claim is justified by an unverified self-citation; and no result reduces by definition to its own inputs. The paper's comparison of executable versus non-executable edits is supported by manual annotation (Table 1), and the explanation-evaluation protocol is calibrated against human annotations (Table 3). The most serious issues are empirical validity concerns: the final 2,121 'inconsistent' summaries are not individually human-validated for factual inconsistency or explanation correctness, and the trivial-edit filter (Prompt F.5) checks only edit-type categories rather than document-grounded inconsistency. Such concerns bear on whether the reported scores measure what the paper claims, but they are not circularity in the derivation-chain sense. The paper also builds on the authors' prior SummEdits and executable-editing work, but those are concrete prior datasets and methods rather than self-referential justifications, so they do not raise the circularity score.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted numerical parameters are introduced; the benchmark's validity rests on a handful of domain assumptions about ground truth and LLM-judge reliability, which are only partially validated by manual annotation.

assumptions (3)

domain assumption The source (document, summary) pairs in SummEdits are trustworthy, and the 2,120 consistent summaries used as negative labels are factually consistent with their documents.
Invoked in Section 4 when the benchmark is assembled from Laban et al. (2023) without re-validating the negative labels; if some negatives are inconsistent, reported detection accuracy is miscalibrated.
domain assumption A substring replacement generated under the executable-edit prompt, with an explanation and not filtered as date/number/antonym, reliably produces a summary that is factually inconsistent with the document.
The pipeline in Section 4 and prompts F.1/F.5 assume LLM-generated edits are valid errors; human annotation covers only a small subset, and explanation-quality agreement is low (Kappa 0.49).
domain assumption The LLM-as-judge explanation scores, especially EvalV4 with GPT4o, are a valid proxy for human explanation quality across the full benchmark.
Section 5.2 validates on manually annotated explanations with correlations up to 0.833, but the judge is then applied to all samples without further human checks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits." pith.science (2026). https://pith.science/paper/MVL6Q7EF

@misc{pith2026241213378,
  author       = {Pith},
  title        = {Pith review of: SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MVL6Q7EF}},
  note         = {Machine review of arXiv:2412.13378}
}

read the original abstract

Detecting factual inconsistencies in summarization is critical, yet existing benchmarks lack the necessary challenge and interpretability for robust evaluation. In this paper, we introduce SummExecEdit, a novel pipeline and benchmark leveraging executable edits to assess models on their ability to both detect factual errors and provide accurate explanations. The top-performing model, Claude3-Opus, achieves a joint detection and explanation score of only 0.49 in our benchmark, with individual scores of 0.67 for detection and 0.73 for explanation. We conduct detailed evaluations to assess the current state of models in this field and find that more than half of the 20+ LLMs in our study struggle with over 30% of the SummExecEdit benchmark. Additionally, we identify four primary types of explanation errors, with 45.4% of them involving a focus on completely unrelated parts of the summary.

Figures

Figures reproduced from arXiv: 2412.13378 by the authors.

**Figure 2.** An example of comparison of inconsistent [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Appendix D - Example 1 - SummExecEdit [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Appendix D - Example 2 - SummExecEdit Example 3 Original Summary: Oman is opening up its airspace to Israeli carriers, including El Al. Previously, the airlines had to fly a roundabout route to parts of Asia, adding hours to journey times. Text to Replace: "Oman is ope…

**Figure 5.** Figure 5: Appendix D - Example 3 - SummExecEdit Example 4 Original Summary: Harry and Jacob discuss their preparedness for an exam that will comprise a combination of multiple choice and open question essay formats, and how knowing the format of the test can influence their stud…

**Figure 6.** Figure 6: Appendix D - Example 4 - SummExecEdit Example 5 Original Summary: The speaker is discussing the topic of nutrition, sharing their experience with following the Zone diet and recommending it as a way to learn about food and develop a better relationship with it. They ex…

**Figure 8.** Figure 8: Two-Column annotation interface with highlights used for annotation [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 1 canonical work pages

[1]

Gurucharan, Kripabandhu Ghosh, and Abhijnan Chakraborty

Garima Chhikara, Anurag Sharma, V. Gurucharan, Kripabandhu Ghosh, and Abhijnan Chakraborty. 2024. https://arxiv.org/abs/2406.15809 Lamsum: Creating extractive summaries of user generated content using llms . Preprint, arXiv:2406.15809

arXiv 2024
[2]

Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://doi.org/10.1162/tacl_a_00373 Summeval: Re-evaluating summarization evaluation . Transactions of the Association for Computational Linguistics, 9:391--409

doi:10.1162/tacl_a_00373 2021
[3]

Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong

Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. https://arxiv.org/abs/2112.08542 Qafacteval: Improved qa-based factual consistency evaluation for summarization . Preprint, arXiv:2112.08542

arXiv 2022
[4]

Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/P19-1213 Ranking generated summaries by correctness: An interesting but challenging application for natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214--...

doi:10.18653/v1/p19-1213 2019
[5]

Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. https://arxiv.org/abs/2305.11171 Trueteacher: Learning factual consistency evaluation with large language models . Preprint, arXiv:2305.11171

arXiv 2023
[6]

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2024. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions . ACM Trans. Inf. Syst. Just Accepted

doi:10.1145/3703155 2024
[7]

Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. https://arxiv.org/abs/2404.01261 Fables: Evaluating faithfulness and content selection in book-length summarization . Preprint, arXiv:2404.01261

arXiv 2024
[8]

Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.600 S umm E dits: Measuring LLM ability at factual reasoning through the lens of summarization . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing...

doi:10.18653/v1/2023.emnlp-main.600 2023

Show all 22 references

[9]

Bennett, and Marti A

Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. https://doi.org/10.1162/tacl_a_00453 Summac: Re-visiting nli-based models for inconsistency detection in summarization . Transactions of the Association for Computational Linguistics, 10:163--177

2022 doi
[10]

Philippe Laban, Jesse Vig, Marti Hearst, Caiming Xiong, and Chien-Sheng Wu. 2024. Beyond the chat: Executable and verifiable text-editing with llms. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1--23

2024
[11]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[12]

Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. https://arxiv.org/abs/2303.15621 Chatgpt as a factual inconsistency evaluator for text summarization . Preprint, arXiv:2303.15621

2023 arXiv
[13]

o rg Schl \

Van Bach Nguyen, Paul Youssef, Christin Seifert, and J \"o rg Schl \"o tterer. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.870 LLM s for generating and evaluating counterfactuals: A comprehensive study . In Findings of the Association for Computational Linguistics: E...

2024 doi
[14]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[15]

Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryściński, Justin F

Liyan Tang, Tanya Goyal, Alexander R. Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryściński, Justin F. Rousseau, and Greg Durrett. 2023. https://arxiv.org/abs/2205.12854 Understanding factual errors in summarization: Errors, summarizers, datasets, error detecto...

2023 arXiv
[16]

Liyan Tang, Philippe Laban, and Greg Durrett. 2024 a . https://arxiv.org/abs/2404.10774 Minicheck: Efficient fact-checking of llms on grounding documents . Preprint, arXiv:2404.10774

2024 arXiv
[17]

Vincent, Yu'an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKeown

Liyan Tang, Igor Shalyminov, Amy Wing mei Wong, Jon Burnsky, Jake W. Vincent, Yu'an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKeown. 2024 b . https://arxiv.org/abs/2402.13249 Tofueval: Evaluating hallucinations of ll...

2024 arXiv
[18]

Jiuding Yang, Hui Liu, Weidong Guo, Zhuwei Rao, Yu Xu, and Di Niu. 2024. https://arxiv.org/abs/2403.07557 Sifid: Reassess summary factual inconsistency detection with llm . Preprint, arXiv:2403.07557

2024 arXiv
[19]

Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.acl-long.634 A lign S core: Evaluating factual consistency with a unified alignment function . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

2023 doi
[20]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Pap...

2023
[21]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[22]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Gurucharan, Kripabandhu Ghosh, and Abhijnan Chakraborty

Garima Chhikara, Anurag Sharma, V. Gurucharan, Kripabandhu Ghosh, and Abhijnan Chakraborty. 2024. https://arxiv.org/abs/2406.15809 Lamsum: Creating extractive summaries of user generated content using llms . Preprint, arXiv:2406.15809

arXiv 2024

[2] [2]

Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://doi.org/10.1162/tacl_a_00373 Summeval: Re-evaluating summarization evaluation . Transactions of the Association for Computational Linguistics, 9:391--409

doi:10.1162/tacl_a_00373 2021

[3] [3]

Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong

Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. https://arxiv.org/abs/2112.08542 Qafacteval: Improved qa-based factual consistency evaluation for summarization . Preprint, arXiv:2112.08542

arXiv 2022

[4] [4]

Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/P19-1213 Ranking generated summaries by correctness: An interesting but challenging application for natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214--...

doi:10.18653/v1/p19-1213 2019

[5] [5]

Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. https://arxiv.org/abs/2305.11171 Trueteacher: Learning factual consistency evaluation with large language models . Preprint, arXiv:2305.11171

arXiv 2023

[6] [6]

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2024. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions . ACM Trans. Inf. Syst. Just Accepted

doi:10.1145/3703155 2024

[7] [7]

Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. https://arxiv.org/abs/2404.01261 Fables: Evaluating faithfulness and content selection in book-length summarization . Preprint, arXiv:2404.01261

arXiv 2024

[8] [8]

Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.600 S umm E dits: Measuring LLM ability at factual reasoning through the lens of summarization . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing...

doi:10.18653/v1/2023.emnlp-main.600 2023