Pith. sign in

REVIEW 2 major objections 6 minor 44 references

Long-Form Information Alignment Evaluation Beyond Atomic Facts

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that reordering true facts into deceptive narratives defeats current factuality evaluators, and that event-order verification restores detection.

desk verdict The benchmark is a genuinely useful new attack set; DoveScore is plausible but under-validated, and the sorter's target-order semantics need direct evidence. read the letter →

arxiv 2505.15792 v1 pith:WB44C27L submitted 2025-05-21 cs.CL cs.AIcs.LG

Danna Zheng , Mirella Lapata , Jeff Z. Pan This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords informationalignmentfactualityevaluationmontagelieeventorderconsistencylong-formsummarizationhallucinationdetectionDoveScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most fine-grained factuality evaluators score each atomic fact independently, so they cannot detect a text composed entirely of true statements whose order has been rearranged to imply a false causal story. The paper calls these reordered narratives 'montage lies', builds a benchmark called MontageLie from long-form summaries, and shows that both coarse-grained LLM judges and fine-grained fact-checkers score below 65% AUC-ROC on them, with FactScore at 50.45%. It then introduces DoveScore, which verifies individual facts and also checks whether the events appear in the same order in the source and the target. With that extra event-order signal, DoveScore reaches 65.25% average AUC-ROC, beating existing fine-grained evaluators by more than 8%. That matters because it identifies a realistic failure mode for trustworthy LLM deployment: a system can be accepted or rejected based on whether the evaluator sees beyond atomic facts to the sequence of events.

What carries the argument

The load-bearing object is the event-order score $S_{EO} = 1 - \operatorname{ShuffleD}(\operatorname{Sorted}(F_E^c, s), \operatorname{Sorted}(F_E^c, t))$, where $F_E^c$ is the set of event facts that pass factual verification, $\operatorname{ShuffleD}$ is the normalized inversion count between the source's chronological order and the target's stated order, and $s$ and $t$ are the source and target texts. The sorter component of DoveScore produces the two orderings; the fact checker filters the facts; the decomposer separates order-dependent event facts from order-independent descriptive facts. The final score is $\alpha \cdot S_E \cdot S_{EO} + (1-\alpha) \cdot S_D$, with $\alpha = |F_E|/(|F_E|+|F_D|)$ and with $S_E$ and $S_D$ the fractions of event and descriptive facts that pass verification, so event-heavy texts are judged more by whether their events are in the right order. MontageLie's difficulty levels are defined by the same inversion-count quantity, so the benchmark and the metric share a common scale and the reported 65.25% is the average over those levels.

What would settle it

Take the MontageLie benchmark's known event lists, the original order and the shuffled order used to create each lie, and replace DoveScore's learned sorter with those ground-truth lists before recomputing the event-order score. If average AUC-ROC stays near 65.25%, the sorter's order reconstruction is already near-oracle; if it rises substantially, the method's success depends on the sorter reliably reading the target's stated order, which the paper does not directly validate.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that information alignment is not exhausted by the truth of the parts: when individually correct event facts are reordered relative to the source, the target can still be misaligned because it reverses implied causality, and every existing evaluator, from ROUGE to LLM-as-judge to atomic-fact methods such as FactScore, fails to penalize this reliably. To make the vulnerability measurable, the paper contributes MontageLie, a benchmark of 1,303 narrative instances from TV-recap and book-summary data in which each lie is generated by shuffling a correct summary's event list with a controlled inversion count, at four difficulty levels. To make the failure fixable, it contributes DoveScore, a fine-grained evaluator that separates descriptive facts from event facts, verifies both against the source, and then uses a sorting step to compare the order of correct event facts in the source with their order in the target; the resulting event-order score is merged with factuality scores into one weighted precision measure. The paper reports that this order-aware score raises average AUC-ROC from 50.45% (FactScore) to 65.25%, while the strongest coarse model reaches 64.23%.

Load-bearing premise

The load-bearing premise is that the sorter recovers the order in which the target text itself narrates the events; if it instead reconstructs the true chronological order from world knowledge, a shuffled lie and the truthful summary would receive the same event-order score and DoveScore's advantage would disappear.

Editorial extensions

If this is right

  • Systems built on atomic-fact-only evaluators will accept reordered narratives, so production factuality checks should treat a high atomic-fact score as necessary but not sufficient for alignment.
  • Adding an event-order verification component to FactScore-like pipelines closes most of the measured gap: DoveScore improves from 50.45% to 65.25% average AUC-ROC on MontageLie.
  • MontageLie offers a controlled testbed with four difficulty tiers, so an evaluator's score at the extreme-hard tier, where the shuffle degree is 0.05 to 0.15, directly measures sensitivity to near-truthful causal distortion.
  • Because DoveScore is modular, substituting stronger decomposers, fact checkers, or sorters can transfer improvements to the overall score without redesigning the framework.
  • Coarse-grained LLM judges also fall below 65% even with long-context models, so order-aware prompting or structured verification will be needed for general-purpose evaluators as well.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The montage-lie attack is not limited to summaries: any pipeline that narrates a chronology, such as news recaps, timeline generation, or clinical summaries, could be silently flipped in meaning by reordering true events, so the benchmark's threat model is broader than long-form summarization.
  • A natural next test is an oracle-sorter variant: rerun DoveScore with the known ground-truth event orders from MontageLie; if accuracy jumps well above 65.25%, the remaining weakness is the sorter's ability to read order, not the idea of order verification itself.
  • Lies that keep both facts and order intact but insert causal connectives or emphasis, such as placing 'because' between two true events, would evade DoveScore entirely because its order score would see no change; detecting that style would require explicit causal-relation verification.
  • The event-order score could be used at generation time rather than only as a post-hoc metric: decoding strategies that penalize outputs whose event order deviates from a retrieved source could reduce montage-style hallucinations before a user reads them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper addresses a real weakness in fine-grained factuality evaluators: they verify atomic facts independently and therefore miss 'montage lies' in which every individual statement is true but the narrative order is manipulated to imply misleading causality. The authors introduce MontageLie, a benchmark of such lies built from long-form summaries with controlled event shuffling, and propose DoveScore, a framework that decomposes target text into event and descriptive facts, verifies each fact against the source, and adds an Event Order Score (SEO) that compares the order of verified events in the source and target. Experiments report that existing coarse- and fine-grained evaluators achieve AUC-ROC below 65%, while DoveScore reaches 65.25%, outperforming the best fine-grained baseline by over 8% and FactScore (same backbone) by 14.8%. Code and data are released.

Significance. If the results were solid, the paper would make two useful contributions: a challenging benchmark that exposes a real failure mode of current evaluators, and a framework with a plausible mechanism for detecting order-level distortions. The MontageLie construction is careful (incremental generation, human quality checks, controlled inversion counts), and the authors provide detailed prompts and code, aiding reproducibility. The reported gap between DoveScore and fact-only baselines is large and the task is genuinely novel. However, the central mechanism of DoveScore—the sorter that is supposed to recover the target's narrative order—appears to be implemented in a way that recovers true chronology instead, which would make SEO vacuous. This is a load-bearing issue that needs to be resolved before the claims can be accepted.

major comments (2)
  1. [§5.1, Figure 3, Table 9] The Sorter is prompted to 'determine the correct chronological order of events in a paragraph' (Table 9), and its Example 2 explicitly reorders a flashback paragraph into the true chronological order. In Figure 3, for the lie target 'Amy went to the movies with John. Mike hit Amy. Mike and Amy broke up.', the 'Sort based on TARGET text' output is 'Mike hit Amy. Mike and Amy broke up. Amy went to the movies with John.'—that is the source chronology, not the target's stated order. Consequently, for both truthful and montage-lie targets, Sorted(F_c_E, t) will coincide with Sorted(F_c_E, s), making SEO = 1 in both cases and neutralizing the event-order component. The paper provides no validation that the sorter follows the target's narrative order, and the claimed contribution of SEO (Section 5.3, Figure 10) is not supported by an ablation. The authors should either re-prompt the sorter to order events as they appear in the target paragraph and verify this behavior on lie instances, or provide an explicit ablation showing that the reported improvement comes from SEO; otherwise, DoveScore reduces to a weighted fact-precision score with no order verification.
  2. [§5.3, Table 4] DoveScore's average advantage over the best coarse-grained model (gpt-4o-mini) is only 1.02 points, and on the extreme-hard difficulty level DoveScore is worse (57.87 vs 59.24). This undercuts the claim of consistent robustness across difficulty levels. In addition, the paper reports no confidence intervals, significance tests, or variance estimates for any AUC-ROC comparison, despite having 1,303 instances. The authors should report bootstrap CIs and assess whether the differences (especially the 8% improvement over fine-grained baselines) are statistically significant. This is needed to support the central empirical claim.
minor comments (6)
  1. [Section 3.1.2] The difficulty intervals for ShuffleD (e.g., [0.80, 0.90], [0.55, 0.65]) are described as disjoint but no sensitivity analysis is given; the benchmark's difficulty taxonomy would be stronger with an analysis of how the choice of intervals affects evaluator scores.
  2. [Table 9, Example 2] The sorter prompt's stated goal and its example conflict: the task says 'correct chronological order of events' but the input example is a flashback, and the output is the true chronology, not the order 'based on the paragraph below' as the final instruction states. This ambiguity should be resolved.
  3. [Figure 3] The 'Sort based on TARGET text' output does not match the target's surface order in the lie example; this should be corrected or explained, since it is the only direct illustration of the sorter's behavior.
  4. [Section 4.3 / Table 4] The claim of 'over 8%' improvement is stated relative to the best existing fine-grained method at each difficulty level, but the comparison in Table 3 shows different baselines; a direct comparison against each individual fine-grained method's average AUC would be clearer.
  5. [Section 5.3 / Figure 10] The paper attributes the improvement to SEO based on score distributions, but does not report DoveScore's AUC on the paraphrase subset separately. Since paraphrases include flashback, interjection, and other narrative techniques, the sorter's behavior on these is essential to support the robustness claim.
  6. [Throughout] Several typos and citation issues appear: 'A VERITEC' should be 'AVERITEC' (Section 2.2), 'MONTAGE LIES' appears in Section 4.3, 'A VG' in Table 3, and the reference to 'Huang et al. 2025' for a survey on hallucination should be checked for correctness.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: MontageLie and DoveScore share the same order-distortion measure by design, and the sorter's true-chronology prompt is a validity risk rather than a circular step.

full rationale

The claimed derivation is not circular. MontageLie labels are created by sampling permutations F of an event list E with a controlled inversion count (Eq. 2), and DoveScore's only order-sensitive term is SEO = 1 - ShuffleD(Sorted(F_c_E,s), Sorted(F_c_E,t)) (Eq. 5). Both use the same combinatorial quantity, but the benchmark labels are generated before and independently of DoveScore, the difficulty intervals are hand-set, alpha is a frequency weight rather than a fitted parameter, and no model component is optimized against the test set. The 65.25% vs 50.45% result is an empirical outcome that could have been different, so it is not forced by construction. The one passage that warrants explicit flagging is the Sorter prompt in Table 9, which asks the model to 'determine the correct chronological order of events in a paragraph' rather than to recover the order in which the target presents them; Figure 3's 'Sort based on TARGET text' output for the lie is the source chronology, not the target's stated order. If the sorter consistently behaves this way, Sorted(F_c_E,t) and Sorted(F_c_E,s) would coincide for truthful and montage targets, making SEO uninformative and undermining the paper's attribution of its 14.8% gain to event-order verification. That is a correctness and validation gap (the Limitations section acknowledges only the sorter's efficiency/accuracy trade-off, not this target-order ambiguity), not a circularity of derivation: the equations do not define the target-order sorter output as the source order, and the benchmark itself still has independent content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central method has no fitted parameters; its design relies on several domain assumptions about LLM behavior and about the definition of alignment. The benchmark difficulty thresholds are hand-set. No physical or mathematical entities are introduced.

free parameters (2)
  • MontageLie difficulty intervals = easy [0.80,0.90], medium [0.55,0.65], hard [0.30,0.40], extreme [0.05,0.15]
    Hand-chosen thresholds on ShuffleD that define benchmark difficulty levels; they are not fitted to evaluation outcome.
  • DoveScore weighting alpha = |FE| / (|FE| + |FD|) per target text
    A design choice for combining event and descriptive scores; computed from decomposition rather than optimized on data.
assumptions (4)
  • domain assumption Information alignment should include event-order consistency even when all atomic facts are supported.
    This underlies the entire MontageLie gold labeling and is stated in Sections 1 and 3 as the motivation for the benchmark.
  • domain assumption The LLM sorter follows the target text's narrative event order rather than reconstructing the true chronology from world knowledge.
    This is required for SEO to measure the montage manipulation; no experiments validate which behavior the sorter exhibits (Section 5.1, Table 9).
  • domain assumption LLM decomposition and fact checking in DoveScore are sufficiently accurate in the zero-shot setting.
    No accuracy numbers are reported for the decomposer or fact checker; errors propagate to the final score (Section 5.1).
  • domain assumption The 50-instance human evaluation validates the full MontageLie dataset.
    The benchmark quality check samples 50 instances; the full 1303 instances are not individually validated (Section 3.2.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Long-Form Information Alignment Evaluation Beyond Atomic Facts." pith.science (2026). https://pith.science/paper/WB44C27L

@misc{pith2026250515792,
  author       = {Pith},
  title        = {Pith review of: Long-Form Information Alignment Evaluation Beyond Atomic Facts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WB44C27L}},
  note         = {Machine review of arXiv:2505.15792}
}
read the original abstract

Information alignment evaluators are vital for various NLG evaluation tasks and trustworthy LLM deployment, reducing hallucinations and enhancing user trust. Current fine-grained methods, like FactScore, verify facts individually but neglect inter-fact dependencies, enabling subtle vulnerabilities. In this work, we introduce MontageLie, a challenging benchmark that constructs deceptive narratives by "montaging" truthful statements without introducing explicit hallucinations. We demonstrate that both coarse-grained LLM-based evaluators and current fine-grained frameworks are susceptible to this attack, with AUC-ROC scores falling below 65%. To enable more robust fine-grained evaluation, we propose DoveScore, a novel framework that jointly verifies factual accuracy and event-order consistency. By modeling inter-fact relationships, DoveScore outperforms existing fine-grained methods by over 8%, providing a more robust solution for long-form text alignment evaluation. Our code and datasets are available at https://github.com/dannalily/DoveScore.

Figures

Figures reproduced from arXiv: 2505.15792 by the authors.

Figure 1
Figure 1. The figure illustrates the limitation of existing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Violin plots of scores from gpt-4o-mini on MONTAGELIE. The similar distributions for original and rephrased targets indicate robustness to rephrasing. Comparable trends are observed for other evaluators (see Appendix C). 4.3 Results [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The illustration of DoveScore which includes three core components: the Decomposer, the Fact Checker, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Score Distribution Comparison of Fine-grained Evaluators. SummaC exhibits a similar pattern to [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of words length in MontageLie benchmark. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Distribution of number of event decomposed [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Distribution of Inversion Count Sampled in [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Violin Plots of Score Obtained By Coarse-Grained Evaluators on M [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Violin Plots of Score Obtained By Fine-Grained Evaluators on M [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Violin Plots of SubScores obtained by DOVESCORE on MONTAGELIE. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 9 canonical work pages

  1. [1]

    Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. https://doi.org/10.18653/v1/2021.fever-1.1 The fact extraction and VER ification over unstructured and structured information ( FEVEROUS ) shared task . In Proceedings of the Fourth Workshop on Fact Extrac...

    doi:10.18653/v1/2021.fever-1.1 2021

  2. [2]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=hSyW5go0v8 Self- RAG : Learning to retrieve, generate, and critique through self-reflection . In The Twelfth International Conference on Learning Representations

    2024

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

    2005

  4. [4]

    Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. https://doi.org/10.18653/v1/2022.acl-long.589 S umm S creen: A dataset for abstractive screenplay summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8602--8615, Dublin, Ireland. Association for Computational L...

    doi:10.18653/v1/2022.acl-long.589 2022

  5. [5]

    Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.599 Compression, transduction, and creation: A unified framework for evaluating natural language generation . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580--7605, Online and Punta Can...

    work page doi:10.18653/v1/2021.emnlp-main.599 2021

  6. [6]

    Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. https://doi.org/10.18653/v1/2022.naacl-main.187 QAF act E val: Improved QA -based factual consistency evaluation for summarization . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages ...

    doi:10.18653/v1/2022.naacl-main.187 2022

  7. [7]

    Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. https://doi.org/10.18653/v1/2022.dialdoc-1.19 TRUE : Re-evaluating factual consistency evaluation . In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversatio...

    work page doi:10.18653/v1/2022.dialdoc-1.19 2022

  8. [8]

    Beizhe Hu, Qiang Sheng, Juan Cao, Yang Li, and Danding Wang. 2025. Llm-generated fake news induces truth decay in news ecosystem: A case study on neural news recommendation. arXiv preprint arXiv:2504.20013

    arXiv 2025

Show all 44 references
  1. [9]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and o...

    2025 doi

  2. [10]

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.123 Towards mitigating LLM hallucination via self reflection . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827-...

    2023 doi

  3. [11]

    Jinhwa Kim, Ali Derakhshan, and Ian Harris. 2024. https://doi.org/10.18653/v1/2024.woah-1.12 Robust safety classifier against jailbreaking attacks: Adversarial prompt shield . In Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), pages 159--170, Mexico City...

    2024 doi

  4. [12]

    Donald E Knuth. 1998. The Art of Computer Programming: Sorting and Searching, volume 3. Addison-Wesley Professional

    1998

  5. [13]

    Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.750 Evaluating the factual consistency of abstractive text summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro...

    2020 doi

  6. [14]

    Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.488 BOOKSUM : A collection of datasets for long-form narrative summarization . In Findings of the Association for Computational Lingu...

    2022 doi

  7. [15]

    Bennett, and Marti A

    Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. https://doi.org/10.1162/tacl_a_00453 S umma C : Re-visiting NLI -based models for inconsistency detection in summarization . Transactions of the Association for Computational Linguistics, 10:163--177

    2022 doi

  8. [16]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  9. [17]

    Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, and 1 others. 2025. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407

    2025

  10. [18]

    Nelson Liu, Tianyi Zhang, and Percy Liang. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.467 Evaluating verifiability in generative search engines . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7001--7025, Singapore. Association fo...

    2023 doi

  11. [19]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511--2522

    2023

  12. [20]

    Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization. arXiv preprint arXiv:2303.15621

    2023 arXiv

  13. [21]

    Huanhuan Ma, Weizhi Xu, Yifan Wei, Liuji Chen, Liang Wang, Qiang Liu, Shu Wu, and Liang Wang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.556 EX - FEVER : A dataset for multi-hop explainable fact verification . In Findings of the Association for Computational Linguisti...

    2024 doi

  14. [22]

    Dominik Macko, Aashish Anantha Ramakrishnan, Jason Samuel Lucas, Robert Moro, Ivan Srba, Adaku Uchendu, and Dongwon Lee. 2025. Beyond speculation: Measuring the growing presence of llm-generated texts in multilingual disinformation. arXiv preprint arXiv:2503.23242

    2025

  15. [23]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. https://openreview.net/forum?id=RwzFNbJ3Ez Selfcheck GPT : Zero-resource black-box hallucination detection for generative large language models . In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  16. [24]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  17. [25]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  18. [26]

    Nirmal Roy, Leonardo F. R. Ribeiro, Rexhina Blloshmi, and Kevin Small. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.622 Learning when to retrieve, what to rewrite, and how to respond in conversational QA . In Findings of the Association for Computational Linguistics: ...

    2024 doi

  19. [27]

    Michael Sejr Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. https://openreview.net/forum?id=fKzSz0oyaI AV eritec: A dataset for real-world claim verification with evidence from the web . In Thirty-seventh Conference on Neural Information Processing Systems Datasets an...

    2023

  20. [28]

    Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.529 Q uest E val: Summarization asks for fact-based evaluation . In Proceedings of the 2021 Conference o...

    2021 doi

  21. [29]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  22. [30]

    Jiasheng Si, Yibo Zhao, Yingjie Zhu, Haiyang Zhu, Wenpeng Lu, and Deyu Zhou. 2024. https://doi.org/10.18653/v1/2024.acl-long.835 CHECKWHY : Causal fact verification via argument structure . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    2024 doi

  23. [31]

    Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.51 F ine S ur E : Fine-grained summarization evaluation using LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    2024 doi

  24. [32]

    Yixiao Song, Yekyung Kim, and Mohit Iyyer. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.552 V eri S core: Evaluating the factuality of verifiable claims in long-form text generation . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9...

    2024 doi

  25. [33]

    Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. https://doi.org/10.18653/v1/2023.acl-long.650 Understanding factual errors in summarization: Errors, summarizers, datasets, error detec...

    2023 doi

  26. [34]

    Liyan Tang, Philippe Laban, and Greg Durrett. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.499 M ini C heck: Efficient fact-checking of LLM s on grounding documents . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818-...

    2024 doi

  27. [35]

    Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu ' an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKeown. 2024 b . https://doi.org/10.18653/v1/2024.naacl-long.251 T ofu E val: Evaluating hallucinatio...

    2024 doi

  28. [36]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

    2018 doi

  29. [37]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Zixia Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V Le. 2024. https://openreview.net/forum?id=4M9f8VMt2C Long-form factuality in large language models . In The Thirty-eighth Annual Co...

    2024

  30. [38]

    Junjie Ye, Guanyu Li, SongYang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Tao Ji, Qi Zhang, Tao Gui, and Xuanjing Huang. 2025. https://aclanthology.org/2025.coling-main.12/ T ool E yes: Fine-grained evaluation for tool learning capabilities of large l...

    2025

  31. [39]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://openreview.net/forum?id=5Ya8PbvpZ9 BARTS core: Evaluating generated text as text generation . In Advances in Neural Information Processing Systems

    2021

  32. [40]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.acl-long.634 A lign S core: Evaluating factual consistency with a unified alignment function . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023 doi

  33. [41]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations

    2020

  34. [42]

    Yue Zhang, Jingxuan Zuo, and Liqiang Jing. 2024. Fine-grained and explainable factuality evaluation for multimodal summarization. arXiv preprint arXiv:2402.11414

    2024

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.