Pith. sign in

REVIEW 2 major objections 4 minor 24 references

MSA at SemEval-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detection

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a prompt-based ensemble of four LLMs, with one model extracting candidate hallucination spans and three voting on them via probability scores, can rank first in Arabic and Basque and reach the top three in eight of eleven…

desk verdict A credible ensemble-voting recipe with top SemEval ranks, but the printed algorithm needs ground truth at inference, so the results are not reproducible as written. read the letter →

arxiv 2505.20880 v1 pith:O2LPJLFZ submitted 2025-05-27 cs.CL

Baraa Hikal , Ahmed Nasreldin , Ali Hamdi This is my paper
classification cs.CL
keywords hallucinationdetectionmultilingualNLPLLMensembleweaklabelingfuzzymatchingspanextractionMu-SHROOMSemEval-2025Task3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a training-free pipeline can locate hallucinated spans in LLM-generated text across a range of languages. The pipeline has three stages: a task-specific prompt makes one LLM extract candidate spans; three other LLMs independently score each span and the probabilities are averaged, with a 0.7 threshold deciding what counts as a hallucination; and Levenshtein-based fuzzy matching aligns the spans to human-style boundaries. The authors report that this recipe ranked first in Arabic and Basque, second in German, Swedish, and Finnish, and third in Czech, Farsi, and French among the Mu-SHROOM shared-task entries. If these results hold, a multilingual hallucination detector can be assembled from off-the-shelf LLMs and prompts rather than trained from scratch.

What carries the argument

The load-bearing mechanism is the rotate-and-vote ensemble: at each of four runs, one LLM is the span extractor and the other three are adjudicators that return hallucination probabilities $p_{ij} \in [0,1]$ for each candidate span; the probabilities are averaged across the three adjudicators, and a span is labelled hallucinated when its mean score in a run is at least $\tau = 0.7$, with a majority vote across the four runs deciding the final label. The fuzzy-matching step then computes a Levenshtein-based similarity between each predicted span and the target span, keeping aligned spans when the partial-ratio similarity is at least 0.9. This post-processing is what converts raw LLM extractions into character-accurate span boundaries.

What would settle it

Run the pipeline on the official test set without ever passing the gold spans into the fuzzy-matching step; if the reported IoU scores in Table 5 cannot be reproduced, the public algorithm description is incomplete or the scores depend on test-time ground truth.

Watch

Extended reading notes

Core claim

The central discovery is that an ensemble of four LLMs—Gemini-2.0-Flash-Exp, Qwen-2.5-Max, GPT-4o, and DeepSeek-V3—can act as a self-adjudicating committee for hallucination span detection. Each model takes a turn as the span extractor while the other three vote on the hallucination probability of every candidate span, and a span is kept when its averaged probability is at least 0.7. The authors find that this simulation of the human annotation process, combined with a fuzzy-matching post-processing step, produces top-three rankings in eight of the eleven evaluated languages. They interpret the results as evidence that ensemble verification reduces model bias and that language-specific differences in which extractor works best point to the value of per-language model selection.

Load-bearing premise

The load-bearing premise is that the fuzzy-matching refinement step in Algorithm 1 can be applied without access to ground-truth spans, yet the pseudocode passes a variable called `Ground Truth` directly into that step, while the test set is described as unlabeled.

Editorial extensions

If this is right

  • A detector built from four off-the-shelf LLMs and a prompt could replace fine-tuned hallucination classifiers, especially in languages where labeled training data is scarce.
  • Because the best extractor varies by language (Gemini for most, DeepSeek for German and Finnish, GPT-4o for Italian, Qwen for Hindi), a language-aware model selector is a natural next step.
  • The 0.7 probability threshold, tuned on validation data, appears to transfer across languages, suggesting annotators' soft labels are comparable even when languages differ.
  • The soft probabilities produced by the adjudicators can be used directly as confidence scores, not just to make hard span decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to re-run the pipeline on the official test set with the fuzzy-matching step removed; the resulting drop in IoU would quantify how much of the reported ranking comes from span alignment rather than from the ensemble voting.
  • The rotate-and-vote design is essentially a multi-annotator agreement scheme; the same machinery could be applied to generate consensus hallucination labels for new unannotated corpora.
  • The paper's per-language best-extractor pattern suggests that an ensemble could be made cost-aware: query a cheap model for easy languages and a stronger model only where the cheap one is uncertain, rather than rotating all four everywhere.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper describes a system submitted to SemEval-2025 Task 3 (Mu-SHROOM) for multilingual hallucinated-span detection. The pipeline consists of a prompt-engineered weak-label generation step, an LLM ensemble in which one of four models extracts hallucination spans and the other three adjudicate through probability-based voting, a 0.7 consensus threshold tuned on validation, and a fuzzy-matching post-processing step. The authors report per-language IoU and probability-correlation scores for each model-as-extractor combination and a final per-language ranking, claiming 1st place in Arabic and Basque, 2nd in German, Swedish, and Finnish, and 3rd in Czech, Farsi, and French.

Significance. If the described pipeline were actually executable on unlabeled test data, the reported top rankings would be a useful demonstration that LLM ensemble verification plus lightweight span refinement is competitive for multilingual hallucination detection. The paper has some genuine strengths: the threshold selection is performed on validation and applied to test, which is standard hyperparameter tuning and not circular; the model-rotation design is clearly presented; and per-language results are reported in full. However, the central claim is currently not reproducible from the manuscript because the printed algorithm consumes ground-truth spans at inference, and the per-language extractor-selection protocol is unspecified. Given that the rankings are the paper's primary evidence, these issues are load-bearing.

major comments (2)
  1. [Appendix B, Algorithm 1 (line 13); Section 3.4; Section 4.1] Algorithm 1 requires Ground Truth as an input to FuzzyMatch(S', Ground Truth, 0.9), and Section 3.4 defines the similarity score against ground-truth spans g_j. Section 4.1 states that the test set was unlabeled and that evaluation was performed by the task organizers. As written, the pipeline cannot be executed on the test set, so the test-set results in Tables 1–5 cannot have been produced by the described method. The paper must either provide a test-time refinement procedure that does not use gold spans, or explicitly describe a different post-processing step that was actually applied. This is not a peripheral presentation issue; it invalidates the reproducibility of the reported rankings.
  2. [Section 4.3, Table 5] The protocol for selecting the per-language 'best' span extractor is not stated. Table 5 reports, for each language, a single span extractor and the resulting rank. If the extractor was chosen after inspecting test-set IoU or Corr values, the final ranks are optimistically biased and the comparison to other systems is inappropriate. If it was chosen via a validation-set criterion, that criterion and threshold must be specified. This selection step is essential to the headline claim of top rankings and needs a precise, reproducible description.
minor comments (4)
  1. [Abstract and Section 5] There are minor typographical issues: '2 nd' in the abstract should be '2nd', and 'forSemEval' in Section 5 is missing a space.
  2. [Section 3.4] The text says fuzzy matching uses a partial-ratio similarity with threshold 0.9, but the formula given is full Levenshtein similarity. These are not equivalent, and the definition should be reconciled so the refinement step is unambiguous.
  3. [Algorithm 1 (Appendix B)] In line 4, the notation p_ij does not explicitly state what i and j index; it would help to define that i indexes the candidate span and j indexes the three adjudicator models in the current run.
  4. [Table 5 caption] The caption says 'final rank' but does not state whether the rank is based on IoU, probability correlation, or an official combined metric; this should be clarified so the reader can interpret the reported ranks.

Circularity Check

1 steps flagged · score 6.0 of 10

Fuzzy-matching refinement ingests ground truth at inference, so the reported IoU is partly constructed from gold spans; extraction and adjudication are otherwise self-contained.

  1. other [Appendix B, Algorithm 1, line 13 (with Sections 3.4 and 4.2); test-set use in Section 4.1]
    "Apply fuzzy matching for span refinement: S* <- FuzzyMatch(S', Ground Truth, 0.9) ... If Similarity(si, gj) >= 0.9, the span is considered correctly aligned. ... The test set contained unlabeled examples, and final system evaluation was conducted by the task organizers."

    The final step that produces the reported prediction set S* requires gold spans ('Ground Truth') as an input, while the headline metric IoU (Section 4.2) is computed against exactly those gold spans. The predicted output is therefore, in its refinement component, a function of the target data: IoU(S*, G) with S* = FuzzyMatch(S', G, 0.9) is partially forced by construction. Section 4.1 states the test set was unlabeled and scored by the organizers, so Algorithm 1 as printed cannot be executed at test time. Either gold labels were used at inference, which would make the reported test IoU partly self-defined, or the actual test-time refinement differs from the published algorithm.

full rationale

Most of the proposed derivation chain is not circular: the 0.7 threshold is tuned on validation data (Section 3.3), the per-model probability aggregation is a defined ensemble rule, and the LLM choices are based on external leaderboards. These are ordinary hyperparameter choices and model selections. However, Algorithm 1 in Appendix B makes the final output depend on ground truth spans via FuzzyMatch(S', Ground Truth, 0.9). Section 3.4 defines the similarity specifically against ground truth spans g_j, and Section 4.2 defines the reported IoU against gold spans. Thus the final refinement step constructs the prediction from the same labels used to measure it, making the reported test IoU, and hence the rankings built on it, partly self-defined. Section 4.1 states the test set was unlabeled, so the printed algorithm cannot be the test-time procedure; no alternative test-time post-processing is given. This is a load-bearing empirical flaw, but it does not infect the core ensemble-adjudication logic, which remains independently meaningful. The per-language best-extractor selection (Table 5) adds selection bias but is not a derivation-to-fit circularity. Overall, one central reported result reduces by construction, so the circularity score is 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No invented entities; the system uses off-the-shelf LLMs. The central method depends on several hand-set constants and on the assumption that LLM adjudicator probabilities can substitute for human annotation. The fuzzy matching step as written also depends on access to ground truth, which is a flaw rather than a stated axiom.

free parameters (3)
  • hallucination threshold tau = 0.7
    Chosen empirically on the validation set in Section 3.3 to balance precision and recall.
  • fuzzy matching similarity threshold = 0.9
    Set in Section 3.4 for partial ratio matching of predicted spans to golden spans.
  • span extractor per language = Varies by language, Table 5 (e.g., Gemini-2.0-Flash-Exp for AR, EU, SV, CS, FA, FR, EN)
    The reported 'best' configuration is selected from four runs per language; the selection protocol (validation vs test) is not specified, making it a free choice that affects the claimed results.
assumptions (4)
  • domain assumption LLM probability scores are comparable and meaningful for hallucination likelihood
    Section 3.3 defines pij = Mj(si, Q) and treats these scores as valid hallucination probabilities without calibration or cross-model normalization.
  • domain assumption Averaging and majority voting across adjudicator LLMs approximates human annotation workflow
    Section 3.3 states the ensemble is inspired by Mu-SHROOM human adjudication, but no evidence is provided that LLM voting reproduces human agreement.
  • domain assumption Levenshtein similarity at threshold 0.9 correctly aligns predicted spans to gold spans
    Section 3.4 assumes this alignment improves accuracy, but the threshold and metric are not validated against annotation variability.
  • domain assumption Weak labels produced by prompt-engineered LLMs are high quality
    The title and abstract assert 'high quality weak labeling', but the paper does not measure weak label quality except through final task ranks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MSA at SemEval-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detection." pith.science (2026). https://pith.science/paper/O2LPJLFZ

@misc{pith2026250520880,
  author       = {Pith},
  title        = {Pith review of: MSA at SemEval-2025 Task 3: High Quality Weak Labeling and LLM Ensemble Verification for Multilingual Hallucination Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2LPJLFZ}},
  note         = {Machine review of arXiv:2505.20880}
}
read the original abstract

This paper describes our submission for SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. The task involves detecting hallucinated spans in text generated by instruction-tuned Large Language Models (LLMs) across multiple languages. Our approach combines task-specific prompt engineering with an LLM ensemble verification mechanism, where a primary model extracts hallucination spans and three independent LLMs adjudicate their validity through probability-based voting. This framework simulates the human annotation workflow used in the shared task validation and test data. Additionally, fuzzy matching refines span alignment. Our system ranked 1st in Arabic and Basque, 2nd in German, Swedish, and Finnish, and 3rd in Czech, Farsi, and French.

Figures

Figures reproduced from arXiv: 2505.20880 by the authors.

Figure 1
Figure 1. Overview of our hallucination detection pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance rankings of LLMs according to the Vectara Hallucination Leaderboard ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Dataset examples in different languages. The hallucinated span(s) are highlighted. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 12 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mostafa Abdelrahman. 2024. Hallucination in low-resource languages: Amplified risks and mitigation strategies for multilingual llms. Journal of Applied Big Data Analytics, Decision-Making, and Predictive Modelling Systems, 8(12):17--24

    work page 2024

  4. [4]

    Henri Alaharju. 2024. Ensuring performance and reliability in llm-based applications: A case study

    work page 2024

  5. [5]

    Surajit Chaudhuri, Kris Ganjam, Venkatesh Ganti, and Rajeev Motwani. 2003. Robust and efficient fuzzy match for online data cleaning. In Proceedings of the 2003 ACM SIGMOD international conference on Management of data, pages 313--324

    work page 2003

  6. [6]

    Baraa Hikal, Ahmed Nasreldin, Ali Hamdi, and Ammar Mohammed. 2025. Few-shot optimized framework for hallucination detection in resource-limited nlp systems. arXiv preprint arXiv:2501.16616

    work page Pith review arXiv 2025

  7. [7]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1--55

    2025

  8. [8]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1--38

    2023

Show all 24 references
  1. [9]

    Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8:64--77

    2020

  2. [10]

    Haoqiang Kang, Terra Blevins, and Luke Zettlemoyer. 2024 a . Comparing hallucination detection metrics for multilingual generation. arXiv preprint arXiv:2402.10496

    2024 arXiv

  3. [11]

    Inwon Kang, William Van Woensel, and Oshani Seneviratne. 2024 b . Using large language models for generating smart contracts for health insurance from textual policies. In AI for Health Equity and Fairness: Leveraging AI to Address Social Determinants of Health, pages 129--146...

    2024

  4. [12]

    Wenfeng Liang and et al. 2024. Deepseek-v3 technical report. https://arxiv.org/abs/2412.19437. Accessed: 2025-04-26

    2024 arXiv

  5. [13]

    Haoyang Liu and Haohan Wang. 2024. Genotex: A benchmark for evaluating llm-based exploration of gene expression data in alignment with bioinformaticians. arXiv preprint arXiv:2406.15341

    2024 arXiv

  6. [14]

    Rahul Mehta, Andrew Hoblitzell, Jack O’keefe, Hyeju Jang, and Vasudeva Varma. 2024. Halu-nlp at semeval-2024 task 6: Metacheckgpt-a multi-task hallucination detection using llm uncertainty and meta-models. In Proceedings of the 18th International Workshop on Semantic Evaluatio...

    2024

  7. [15]

    a t Munch \

    Ercong Nie. 2022. Zero-shot learning on low-resource languages by cross-lingual retrieval. Masterarbeit im Studiengang Computerlinguistik an der Ludwig-Maximilians-Universit \"a t Munch \"e n Fakult \"a t fur Sprach-und Literaturwissenschaften

    2022

  8. [16]

    OpenAI. 2024. Gpt-4o system card. https://arxiv.org/html/2410.21276v1. Accessed: 2025-04-26

    2024 arXiv

  9. [17]

    Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages...

    2019

  10. [18]

    Leily Sheugh and Sasan H Alizadeh. 2015. A note on pearson correlation coefficient as a metric of similarity in recommender system. In 2015 AI & Robotics (IRANOPEN), pages 1--6. IEEE

    2015

  11. [19]

    Ryan Smith, Jason A Fries, Braden Hancock, and Stephen H Bach. 2024. Language models in the loop: Incorporating prompting into weak supervision. ACM/JMS Journal of Data Science, 1(2):1--30

    2024

  12. [20]

    Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. 2025. Llm-check: Investigating detection of hallucinations in large language models. Advances in Neural Information Processing Systems, 37:34188--34216

    2025

  13. [21]

    Reut Tsarfaty, Djam \'e Seddah, Sandra K \"u bler, and Joakim Nivre. 2013. Parsing morphologically rich languages: Introduction to the special issue. Computational linguistics, 39(1):15--22

    2013

  14. [22]

    Ra\'ul V\'azquez, Timothee Mickus, Elaine Zosa, Teemu Vahtola, J\"org Tiedemann, Aman Sinha, Vincent Segonne, Fernando S\'anchez-Vega, Alessandro Raganato, Jindřich Libovický, Jussi Karlgren, Shaoxiong Ji, Jindřich Helcl, Liane Guillou, Ona de Gibert, Jaione Bengoetxea, Joseph...

    2025

  15. [23]

    Vectara. 2024. https://github.com/vectara/hallucination-leaderboard/commit/9708eccda25bf8640db6c6748ac25369947309ac Hallucination leaderboard . GitHub repository commit

    2024

  16. [24]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.