Pith. sign in

REVIEW 5 major objections 5 minor 29 references

Incongruence Identification in Eyewitness Testimony

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured 6W prompts improve eyewitness incongruence detection by 5.63% F1.

desk verdict New dataset and task framing for eyewitness-testimony incongruence, but the headline F1 gain is undercut by a schema-alignment confound and missing reproducibility details. read the letter →

arxiv 2502.05650 v1 pith:CLI7UTOF submitted 2025-02-08 cs.CL

Akshara Nair , Zeba Afroz , Md Shad Akhtar This is my paper · ORCID
classification cs.CL
keywords incongruencedetectioneyewitnesstestimony6Wpromptingmulti-hopreasoninginstructiontuningspanextractionMINDdatasetcontradiction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that contradictions between two eyewitness accounts can be detected and localized at the level of the exact conflicting phrases, not just classified as true or false. To test this, the authors built MIND, a dataset of 2,979 context-answer pairs from 389 testimonies about 149 crime-related events, with expert annotations marking incongruent spans. They then propose INTEND, an instruction-tuned framework whose 6W prompts (who, what action, what object, when, where, why) ask the model to mark each dimension as agreeing, contradicting, or absent. The headline result is that INTEND improves incongruence detection F1 by +5.63% over the same models with plain question prompts, reaching 0.75 F1 with Mistral-7B, and that its three-hop reasoning pipeline beats few-shot baselines by more than 9% F1 on span identification and alignment. If the claims hold, automated systems can move from flagging that two statements conflict to saying exactly which reported details conflict and why.

What carries the argument

INTEND's two components carry the argument. For detection, the 6W prompt turns the comparison into a cloze-style fill-in-the-mask problem: for each of six dimensions (identity, action, object, timeline, location, reason), the model chooses 'agrees with,' 'contradict,' or 'is absent from' between Witness A and Witness B. For reasoning, a three-hop pipeline first extracts fine-grained key details from each testimony, then infers whether any details conflict and why, then extracts the exact conflicting spans. The MIND dataset supplies the training and evaluation substrate: 2,979 context-answer pairs, 1,850 incongruent and 1,129 non-incongruent, with expert-marked spans averaging 10.34 tokens.

What would settle it

An independent re-annotation of a random sample of MIND pairs, with agreement measured and then INTEND run under its original 6W prompt and under a paraphrased prompt covering the same dimensions, would settle whether the +5.63% gain comes from the 6W structure itself or from reproducing the annotation scheme.

Watch

Extended reading notes

Core claim

The central claim is that a model prompted along six investigative dimensions and made to reason in three explicit hops can find incongruences in paired eyewitness testimonies more accurately than standard question-prompted instruction tuning. The paper reports INTEND achieving 0.75 F1 with Mistral-7B, a +5.63% F1 improvement over the same model prompted with questions, and consistent 3–8% gains across Llama-3, Gemma, and Qwen. For span extraction, the multi-hop setup outperforms few-shot by more than 9% F1 on spans in both testimonies and on the alignment between them. The authors also report that human evaluators preferred the multi-hop outputs over few-shot on clarity, logical exclusivity, context relevance, and coverage.

Load-bearing premise

The whole comparison rests on MIND's gold labels being correct, but the paper reports no inter-annotator agreement for the four annotators who marked incongruent spans, and the 6W prompt was built from the same six dimensions used in that annotation.

Editorial extensions

If this is right

  • Structured 6W prompts improve binary incongruence detection F1 by 3–8% across all four instruction-tuned LLMs tested, with the best result at 0.75 F1.
  • A three-hop reasoning pipeline (key details, reason, span extraction) beats few-shot span generation by more than 9% F1 on both span identification and alignment.
  • Open-weight 7–9B LLMs with INTEND outperform fine-tuned long-context MLMs such as Longformer and Big-Bird on the detection task.
  • Human evaluators rate multi-hop outputs better than few-shot outputs on clarity, logical exclusivity, context relevance, and coverage.
  • The MIND dataset makes span-level incongruence reasoning possible and is split into 65/35 train/test with 1,938 training and 1,049 test pairs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the 6W prompt mirrors the same six dimensions used to create MIND's gold labels, a fair follow-up is to test INTEND on a dataset annotated with a different scheme, or to measure inter-annotator agreement on a re-annotated sample; if those labels are noisy, part of the +5.63% gain may be the model reproducing the annotation schema.
  • Editorial extension: the same 6W-plus-multi-hop recipe could transfer to other settings where two accounts of one episode must be reconciled, such as medical histories, insurance claims, or regulatory interviews, if incongruence definitions are adapted.
  • Editorial extension: in legal or investigative use, the system's role would be to surface candidate conflicting spans for human review, not to decide credibility; the paper's ethical statement says as much, but a concrete human-in-the-loop workflow is not specified.
  • A testable extension: run the three-hop pipeline with the middle hop replaced by an explicit 'no conflict detected' gate before span extraction; the error analyses suggest many false alarms come from hop 2 flagging compatible details as contradictions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces a new task and dataset for detecting incongruence between pairs of eyewitness testimonies, along with an instruction-tuned framework (INTEND) that combines a 6W prompt for binary detection and a three-hop reasoning procedure for span extraction. The authors report that INTEND improves detection F1 by 5.63% over question-prompt baselines on their MIND dataset and that multi-hop span extraction outperforms few-shot baselines by over 9% F1 for the best models. The paper also presents an error analysis and a human evaluation of generated spans.

Significance. If the results hold, this is a useful contribution: MIND is a new multi-perspective dataset with span-level incongruence annotations, and the combination of 6W structured prompts and multi-hop reasoning is a sensible, interpretable approach. The paper includes human evaluation and reports coverage for generated spans, which is a strength. However, the central comparisons rest on several underspecified choices (binary aggregation, training-set sizes, metric definitions) and a potential schema-alignment confound, so the quantitative claims need revision and additional controls before the contribution is fully established.

major comments (5)
  1. [Section 4.2 / Figure 1] The paper never specifies how the six per-dimension mask labels ('agrees', 'contradict', 'absent') are aggregated into the binary detection label reported in Table 4b. For reproducibility of the central +5.63% claim, specify the aggregation rule (e.g., any 'contradict' yields True, or majority voting) and describe how absent labels are treated.
  2. [Section 3.2 / Appendix A] No inter-annotator agreement is reported for the MIND gold annotations, and the six annotation criteria in Section 3.2 and Appendix A are exactly the six dimensions in the INTEND prompt. This creates a schema-alignment confound: the model may be reproducing the annotation rubric rather than discovering incongruence. Please report agreement statistics (e.g., Cohen's kappa or Krippendorff's alpha per dimension and for binary labels), and add a control condition with a prompt that conveys the same comparison instruction without enumerating the six dimensions, or evaluate on an independently labeled subset.
  3. [Section 5.2] The training-set sizes for the comparison are not matched or fully specified. The text states that 151 samples are annotated with the 6W template for INTEND, but does not state how many training samples are used for the question-prompt instruction-tuned LLM baselines. If the question-prompt models use the full 1,938 training pairs while INTEND uses only 151, the comparisons in Table 4b are not controlled; if both use 151, please state so explicitly. This is needed to interpret the headline F1 improvement.
  4. [Section 5.3 / Table 5] The claim that the multi-hop approach 'significantly outperforms the few-shot baselines by a margin of >9%' is not supported by the full table: Qwen-2's multi-hop F1 on T1 (0.395) is below its few-shot F1 (0.447), and Mistral's gain is small. The 9% figure appears to compare Llama-3 multi-hop against the best few-shot result from a different model, confounding model and method. Report matched model-by-model comparisons with significance tests or confidence intervals.
  5. [Section 5.2 / Table 5] The span extraction evaluation is underspecified: the paper does not define how precision, recall, and F1 are computed over spans (token-level? exact segment match?), nor how the few-shot exemplars are selected and how many are used. Without these definitions, the span-level gains and coverage numbers cannot be reproduced or interpreted.
minor comments (5)
  1. [Abstract / Section 1 / Table 2] The abstract states 2,927 pairs while Section 1 and Table 2 state 2,979; correct the inconsistency.
  2. [Throughout] Fix typos: 'Mistal' (Section 5.3), 'Emperical' (Abstract), 'incongrunce' (Section 4.2), 'alignement' (Section 5.3), and 'Empirical' in the abstract.
  3. [Section 5.5] The human evaluation uses only 15 test examples; report the total number of evaluated outputs, the distribution of ratings, and inter-annotator agreement, since percentages in Table 7 are otherwise hard to interpret.
  4. [Section 5.1.1 / Appendix B] Section 5.1.1 says the maximum combined input size is about 600 tokens, but Appendix B sets context length to 1024; clarify whether the 1024 context length is sufficient and whether truncation is used.
  5. [Table 4a] Longformer's question-prompt F1 (0.63) is inconsistent with its reported precision/recall (0.66/0.63); verify the numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: INTEND's gains come from held-out test evaluation; the 6W/annotation overlap is a confound, not a by-construction reduction.

full rationale

Walking the derivation chain, the paper's claims are empirical comparisons on a held-out 35% test split of the new MIND dataset (Section 5.2), not derivations from self-cited results. The +5.63% detection gain and the >9% span/alignment gains are reported against baselines on that split, so they are not forced by the training prompts or by the 151 instruction-tuning samples. The 6W prompt in Figure 1 and the annotation criteria in Appendix A share the same six dimensions (identity, action, object, timeline, location, reason); this is a schema-alignment confound that could inflate the apparent benefit of INTEND, but it is not circular because the model must still infer the mask values from testimony text and the gold labels were produced by human annotators, not by the prompt. No load-bearing self-citations or imported uniqueness theorems appear; external citations (Mott, Fei et al., Wei et al.) support standard techniques. Missing inter-annotator agreement is an annotation-reliability limitation, not a circularity. Therefore no circular step meets the evidentiary bar of Eq. X = Eq. Y or fitted-parameter-renamed-as-prediction.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The central claim depends on data-collection and annotation assumptions rather than physical or mathematical postulates. No new entities are introduced. The main confound is that the 6W prompt mirrors the annotation dimensions.

free parameters (7)
  • Number of reasoning hops = 3
    Selected after comparing one, two, and three hops in Appendix E; the choice is based on observed performance, effectively tuning the method to the test data.
  • Number of 6W-annotated training samples = 151
    Section 5.2 reports that 151 samples were annotated with the 6W template and used as the training set; the relation of this subset to the 1,938-sample training split is not clarified.
  • Temperature = 0.6
    Hand-set generation hyperparameter (Appendix B) affecting all LLM outputs.
  • Top-p = 0.9
    Hand-set generation hyperparameter (Appendix B).
  • Context length = 1024
    Input truncation length for MLM and LLM experiments (Appendix B).
  • Learning rate / epochs / batch size = 2e-4 / 7 / 8
    Fine-tuning hyperparameters (Appendix B) that influence final F1.
  • LoRA alpha = 16
    Parameter-efficient fine-tuning setting (Appendix B) with lora_dropout set to 0.
assumptions (3)
  • domain assumption The wizard-of-oz collection setup produces naturalistic deceptive eyewitness behavior that transfers to real interrogations.
    Section 3.1; the entire dataset's validity rests on this assumption, which is not validated against real court or police testimony.
  • ad hoc to paper Six dimensions (identity, action, object, timeline, location, reason) are sufficient to characterize incongruence in eyewitness testimony.
    Section 4.2 and Figure 1; this is the core modeling premise of INTEND, asserted rather than derived.
  • domain assumption Annotator labels for incongruent spans and binary labels are correct and consistent.
    Section 3.2; no inter-annotator agreement is reported, yet gold labels are treated as ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Incongruence Identification in Eyewitness Testimony." pith.science (2026). https://pith.science/paper/CLI7UTOF

@misc{pith2026250205650,
  author       = {Pith},
  title        = {Pith review of: Incongruence Identification in Eyewitness Testimony},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLI7UTOF}},
  note         = {Machine review of arXiv:2502.05650}
}
read the original abstract

Incongruence detection in eyewitness narratives is critical for understanding the reliability of testimonies, yet traditional approaches often fail to address the nuanced inconsistencies inherent in such accounts. In this paper, we introduce a novel task of incongruence detection in eyewitness testimonies. Given a pair of testimonies containing of multiple pairs of question and answer by two subjects, we identify contextually related incongruence between the two subjects. We also mark the span of incongruences in the utterances. To achieve this, we developed MIND(MultI-EyewitNess Deception) - a comprehensive dataset consisting of 2927 pairs of contextually related answers designed to capture both explicit and implicit contradictions. INstruction - TunEd iNcongruity Detection framework based on 6W and multi-hop reasoning approach, aka. INTEND. Drawing from investigative techniques, INTEND address the task as a close-style problem, contradicting on the who, what, when, where and why aspect of the content. Our findings shows that prompt tuning, especially when utilizing our framework, enhances the detection of incongruences by a margin of +5.63 percent. We compare our approach with multiple fine-tuning and prompt tuning techniques on MLMs and LLMs. Emperical results demonstrate convincing performance improvement in F1-score over fine-tuned and regular prompt-tuning techniques, highlighting the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2502.05650 by the authors.

Figure 1
Figure 1. An example of prompt with 6W s instruction for incongruence detection. T1 and T2. Finally, annotators3 carefully anno￾tate the inconsistent sections with contradicting information. This procedure entailed pinpointing textual spans reflecting the location of inconsisten￾cies, providing a higher degree of information. To maintain uniformity and precision, we train anno￾tators with an annotation guidelines and mandated… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 13 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  4. [4]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    arXiv 2020

  5. [5]

    Erika Doggett and Alejandro Cantarero. 2016. https://doi.org/10.18653/v1/W16-6202 Identifying eyewitness news-worthy events on T witter . In Proceedings of the Fourth International Workshop on Natural Language Processing for Social Media, pages 7--13, Austin, TX, USA. Association for Computational Linguistics

    work page doi:10.18653/v1/w16-6202 2016

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    Hao Fei, Bobo Li, Qian Liu, Lidong Bing, Fei Li, and Tat-Seng Chua. 2023. Reasoning implicit sentiment with chain-of-thought prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1171--1182

    work page 2023

  8. [8]

    Fernandes and Muhammad S

    Sinead V. Fernandes and Muhammad S. Ullah. 2021. https://doi.org/10.1109/IEMCON53756.2021.9623077 Development of spectral speech features for deception detection using neural networks . In 2021 IEEE 12th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pages 0198--0203

    work page arXiv 2021

Show all 29 references
  1. [9]

    Rachel Leigh Greenspan, Alex Lyman, and Paul Heaton. 2024. https://doi.org/10.1177/09567976241229028 Assessing verbal eyewitness confidence statements using natural language processing . Psychological Science, 35(3):277--287. PMID: 38376954

    2024 doi

  2. [10]

    Viresh Gupta, Mohit Agarwal, Manik Arora, Tanmoy Chakraborty, Richa Singh, and Mayank Vatsa. 2019. https://doi.org/10.1109/CVPRW.2019.00016 Bag-of-lies: A multimodal dataset for deception detection . In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Worksh...

    2019

  3. [11]

    Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. 2006. Negation, contrast and contradiction in text processing. In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 1, AAAI'06, page 755–762. AAAI Press

    2006

  4. [12]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  5. [13]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  6. [14]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2022. https://aclanthology.org/2022.coling-1.152 Understanding and improving zero-shot multi-hop reasoning in generative question answering . In Proceedings of the 29th International Conference on Computational Linguist...

    2022

  7. [15]

    Oğuzhan Katar, Dilek Ozkan, GPT, Özal Yildirim, and Rajendra Acharya. 2022. https://doi.org/10.13140/RG.2.2.11949.15844 Evaluation of gpt-3 ai language model in research paper writing

    2022

  8. [16]

    Ala Kharrat, Lobna Hlaoua, and Lotfi Romdhane. 2022. https://doi.org/10.1007/978-3-031-16014-1_19 Contradiction Detection Approach Based on Semantic Relations and Evidence of Uncertainty , pages 232--245. Springer International Publishing

    2022 doi

  9. [17]

    Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. https://doi.org/10.1609/aaai.v32i1.12022 Scitail: A textual entailment dataset from science question answering . Proceedings of the AAAI Conference on Artificial Intelligence, 32(1)

    2018 doi

  10. [18]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. https://arxiv.org/abs/2107.13586 Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing . Preprint, arXiv:2107.13586

    2021 arXiv

  11. [19]

    Ilya Loshchilov and Frank Hutter. 2019. https://arxiv.org/abs/1711.05101 Decoupled weight decay regularization . Preprint, arXiv:1711.05101

    2019 arXiv

  12. [20]

    Marie-Catherine Marneffe, Anna Rafferty, and Christopher Manning. 2008. Finding contradictions in text. In Proceedings of ACL-08: HLT, pages 1039--1047

    2008

  13. [21]

    Frank Luther Mott. 1942. Trends in newspaper content. The Annals of the American Academy of Political and Social Science, 219(1):60--65

    1942

  14. [22]

    Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. https://doi.org/10.18653/v1/2021.findings-acl.366 Prompting contrastive explanations for commonsense reasoning tasks . In Findings of the Association for Computational Li...

    2021 doi

  15. [23]

    Ver \'o nica P \'e rez-Rosas, Mohamed Abouelenien, Rada Mihalcea, Yao Xiao, CJ Linton, and Mihai Burzo. 2015. Verbal and nonverbal clues for real-life deception detection. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2336--2346

    2015

  16. [24]

    Timo Schick and Hinrich Schütze. 2021. https://arxiv.org/abs/2001.07676 Exploiting cloze questions for few shot text classification and natural language inference . Preprint, arXiv:2001.07676

    2021 arXiv

  17. [25]

    Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.28 Stretching sentence-pair NLI models to reason over long documents and clusters . In Findings of the Association for Computational Linguisti...

    2022 doi

  18. [26]

    Sara Sol \`a -Sales, Chiara Alzetta, Carmen Moret-Tatay, and Felice Dell'Orletta. 2023. Analysing deception in witness memory through linguistic styles in spontaneous language. Brain sciences, 13(2):317

    2023

  19. [27]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  20. [28]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  21. [29]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

    2020

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.