TUM-MiKaNi at SemEval-2025 Task 3: Towards Multilingual and Knowledge-Aware Non-factual Hallucination Identification

Ekaterina Gikalo; Georg Groh; Miriam Ansch\"utz; Niklas Herbster

REVIEW 2 major objections 5 minor 29 references

TUM-MiKaNi at SemEval-2025 Task 3: Towards Multilingual and Knowledge-Aware Non-factual Hallucination Identification

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper presents MiKaNi, a token-level hallucination detector that verifies atomic facts against English Wikipedia and fuses the result with a BERT pattern model, yielding top-ten results in eight of fourteen languages.

desk verdict A competent shared-task system paper that does what it claims, with the multilingual knowledge-awareness caveat the authors themselves acknowledge for Chinese. read the letter →

arxiv 2507.00579 v1 pith:UA5IUQU6 submitted 2025-07-01 cs.CL cs.AI

Miriam Anschütz , Ekaterina Gikalo , Niklas Herbster , Georg Groh This is my paper · ORCID

classification cs.CLcs.AI

keywords hallucinationdetectionmultilingualNLPtoken-levelannotationfactverificationWikipediaretrievalBERTsupportvectorregressionMu-SHROOM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most hallucination research targets English, and this paper tries to build a detector that works in fourteen languages at the token level. Its system, MiKaNi, combines a retrieval-based fact-checker that verifies atomic claims against English Wikipedia with a BERT model fine-tuned to recognize hallucination patterns, then fuses the two signals through support vector regression. The paper reports top-ten results in eight of the fourteen languages, including English, and shows the ensemble outperforms the task baselines in every language except Chinese. The practical payoff is a model-agnostic detector that can flag exactly which tokens in an LLM answer are non-factual.

What carries the argument

The load-bearing machinery is the two-submodel ensemble. The retrieval-based fact verification model (RFVM) uses GPT-4o to extract atomic facts, generate Wikipedia search terms, and assign token hallucination probabilities against retrieved evidence, with sentence ranking by BM25 and Maximal Marginal Relevance. The BERT-based model (BM) concatenates a BERT token embedding with a part-of-speech embedding and pushes it through linear layers to predict a per-token hallucination probability. A support vector regression model (SVRM) fuses the RFVM score, the BM score, the BM embedding, part-of-speech tags, and a question-answer entity-match feature into the final score.

What would settle it

Run MiKaNi on factually correct answers written in a language or cultural domain that English Wikipedia covers poorly, with human-verified token labels; if correct tokens are systematically assigned high hallucination scores, the English-Wikipedia retrieval path is the point of failure.

Watch

Extended reading notes

Core claim

MiKaNi's central claim is that a multilingual token-level hallucination detector can be built without any access to the generating model's internals, by combining two complementary evidence sources. The retrieval-based fact verification model decomposes each answer into atomic facts, translates them to English, retrieves and ranks relevant Wikipedia sentences using BM25 and Maximal Marginal Relevance, and asks GPT-4o to assign per-token hallucination probabilities against that evidence. The BERT-based model captures distributional and part-of-speech patterns of hallucinated content, and a support vector regression merges both outputs with linguistic features into a final soft label for every token. The paper reports top-ten IoU ranks in eight of the fourteen Mu-SHROOM languages, and its combined system outperforms all baselines in every language except Chinese, where the mark-all baseline is slightly better.

Load-bearing premise

Everything depends on the assumption that translating every atomic fact into English and checking it against English Wikipedia gives a reliable factuality verdict for answers originally written in any of the fourteen languages.

Editorial extensions

If this is right

MiKaNi can be applied to closed or API-only LLMs because it needs only the question-answer text, not logits or internal states.
The same architecture transfers to languages outside the fourteen task languages as long as a POS tagger and multilingual BERT support them.
Combining the fact-checker with the BERT pattern model improves IoU in most languages over either submodel alone, indicating the two signals are complementary.
Token-level scores can pinpoint which atomic facts in an answer are wrong, enabling fine-grained corrections rather than whole-answer rejection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One testable extension would replace English Wikipedia with a multilingual knowledge base so facts are verified in their original language; the paper's Chinese results suggest translation ambiguity hurts retrieval, so the improvement could be measured directly.
An open ablation is swapping GPT-4o in the fact-checker for a smaller open-source model to see how much of the ensemble's gain depends on the proprietary verifier.
Since the paper finds that numbers, proper nouns, and nouns carry most hallucinations, a variant that invests extra capacity in verifying those categories could be compared against the reported IoU scores.
The portability claim can be stress-tested on languages beyond the fourteen by checking whether performance tracks English Wikipedia coverage, which would show how much of the multilingual gain is architecture versus knowledge-base reach.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid, honest shared-task system description that deserves publication in the workshop proceedings, but it is an engineering contribution, not a research breakthrough. The system works as reported; the main caveats are the ones the authors themselves list.

What is actually new: MiKaNi as a specific configuration — GPT-4o-based atomic fact extraction with translation to English, BM25/MMR retrieval from English Wikipedia, a fine-tuned BERT-POS model, and SVR fusion — is not described in prior work, and the per-language results on Mu-SHROOM are a new empirical data point. The authors compare against the three official baselines and report both IoU and Spearman correlation with ranks. The code and model weights are public.

The paper's strengths: the architecture is clearly explained, the prompts are included in full, the training details are specific enough to re-implement, and the qualitative analysis tries to explain where the system fails. The claim of competitiveness is backed by the official numbers: top-10 in eight of fourteen languages, and above all baselines except Chinese. The data are from the official Mu-SHROOM test set, and the citation pattern is reasonable — the self-cite to Anschütz and Groh 2022 is for the SVR ensemble idea and is on point.

Soft spots, in order. First, the knowledge-aware branch is only as multilingual as English Wikipedia plus GPT-4o's translation. The authors translate every atomic fact to English and retrieve only from English Wikipedia, but they never report retrieval success rates or evidence relevance per language. Appendix A shows translation can break retrieval for Chinese, and Table 2 shows RFVM's Chinese IoU is by far the lowest (0.253). The ensemble still does okay because the BERT model carries the weight. So the 'multilingual and knowledge-aware' headline is only partly supported for some languages; per-language retrieval diagnostics would make the claim fully land. This is the stress-test's main concern and it is valid.

Second, the closed-source GPT-4o dependency is a real reusability limitation, and the authors say so themselves in Section 6. They don't pin the API version or give a commit hash, so exact reproduction is not guaranteed. That is minor for a shared-task paper.

Third, there are no error bars or repeated runs, and the significance of rank differences is not discussed. Common in this genre.

Minor: the appendix heading 'Search Term Generation System Prompt' is duplicated for the hallucination prediction prompt, and Table 4's formatting is broken in the pdf. Annoyances, not substance.

My take: the conditional verdict is about right. The central result holds on its own terms. The English-Wikipedia assumption is an acknowledged design choice, not a hidden flaw, and the authors flag the Chinese case honestly. The paper does not oversell.

Who this is for: people working on multilingual hallucination detection and anyone building on shared-task baselines. It is a useful system description, not a methods paper. A serious referee should engage with it — it belongs in workshop proceedings. I would accept it for peer review and recommend minor revision: add per-language retrieval success rates, pin the GPT-4o version, and fix the appendix typos.

Referee Report

2 major / 5 minor

Summary. This paper describes MiKaNi, the authors' submission to SemEval-2025 Task 3 (Mu-SHROOM), a token-level multilingual hallucination detection shared task covering 14 languages. The system combines a retrieval-based fact verification model (RFVM) that extracts atomic facts with GPT-4o, translates them to English, retrieves evidence from the English Wikipedia, and predicts token-level hallucination scores, with a BERT-based model (BM) fine-tuned on multilingual validation data and enriched with part-of-speech embeddings. The two submodels' outputs are combined by a support vector regression model (SVRM). On the official test set the system reports top-10 ranks in eight languages and outperforms the shared-task baselines in all languages except Chinese. The paper also provides per-language ablations of the RFVM and BM scores, a qualitative error analysis, and public code and model weights.

Significance. If the results are taken at face value, MiKaNi is a competitive model-agnostic detector: it does not require access to the generating model's logits, and it shows substantial cross-lingual generalization, including to test-only languages such as Catalan and Farsi. The official shared-task evaluation provides a credible external benchmark, and the public release of code and weights supports reproducibility and further comparison. However, the distinctive 'knowledge-aware' component is not fully validated: the paper does not measure per-language retrieval coverage or isolate the contribution of Wikipedia evidence from GPT-4o's parametric knowledge, so the reported scores do not yet establish that fact verification against English Wikipedia is the source of the multilingual gains.

major comments (2)

[Sections 3.1.1, 3.4.1, Table 2, Appendix A] The paper's central 'knowledge-aware' claim is conditional on the assumption that atomic facts in all 14 languages, once translated to English, can be verified against English Wikipedia. Section 3.1.1 states that facts are translated into English, and Section 3.4.1 states that all search-term prompting is strictly in English with evidence drawn from the English Wikipedia API. Appendix A explicitly attributes Chinese underperformance to translation difficulties. Yet the paper reports no per-language retrieval success rates, evidence relevance judgments, or translation quality metrics. Table 2 shows that the RFVM is highly variable across languages (e.g., IoU 0.2530 for Chinese versus 0.5298 for Swedish; Cor 0.1756 for Chinese versus 0.4803 for Italian), which is consistent with the hypothesis that the retrieval branch fails for some languages. Without such diagnostics, the statement that MiKaNi is a knowledge-aware detector 'across all languages' (Abstract) is not supported; in low-coverage languages the ensemble may be dominated by the BERT component, and the reported competitiveness does not validate the knowledge-aware mechanism. Please add a per-language analysis of retrieval coverage or an ablation that removes the RFVM and reports the resulting score drop.
[Appendix B.3, Section 3.1] The RFVM is not purely retrieval-based: the hallucination-prediction system prompt in Appendix B.3 explicitly instructs GPT-4o to 'rely on your own knowledge' when Wikipedia facts are insufficient. This means the final predictions in the retrieval branch can be generated from the model's parametric memory rather than from the retrieved evidence. The paper does not report how frequently this fallback is used, nor does it evaluate the retrieval branch in isolation with the fallback disabled. Because the system description in Section 3.1 presents the RFVM as 'leveraging Wikipedia as a factual reference source,' the current evidence does not quantify the actual contribution of the Wikipedia retrieval to the scores in Tables 1-3. I recommend adding either a frequency analysis of fallback usage or an ablation comparing the full prompt against a no-retrieval condition.

minor comments (5)

[Section 3.3] The explanation of the QA-entity feature is confusing: the authors write that this feature 'helps to recall non-hallucinated tokens, as we found that many true hallucinations are named entities.' If named entities are frequently hallucinated, a feature indicating entity overlap with the question would more plausibly flag likely hallucination positions; please clarify the intended mechanism or correct the phrasing.
[Figure 2] The caption contains a typo: 'part-of-speechembedding' should be 'part-of-speech embedding'.
[Appendix B] The second system prompt is headed 'Search Term Generation System Prompt' but it actually describes token-level hallucination prediction; the heading should be corrected to 'Hallucination Prediction System Prompt' to avoid confusion.
[Abstract and Section 5] The abstract and conclusion claim that the system supports languages beyond the fourteen in the shared task, but no experiment or demonstration is provided for any additional language; this claim should either be removed or supported by a small proof-of-concept study.
[Section 3.4.2] Please clarify whether the validation split used for early stopping of the BERT model is a random split of the official validation set, and specify the random seed or otherwise confirm that no hyperparameter selection was performed on the official test set; this would help rule out inadvertent test-set tuning.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the system is evaluated on a held-out shared-task benchmark, and no reported prediction is recomputed from test-fit parameters.

full rationale

The paper's central claim is an empirical one: a two-part pipeline (RFVM + BERT-based model combined by SVR) obtains competitive token-level hallucination scores on the Mu-SHROOM benchmark. The evaluation is against held-out human-annotated test data with official IoU and Spearman correlation metrics, and the submodels and regressor are trained on separate training/validation splits (Sections 3.4.2 and 3.4.3). No 'prediction' is derived by reusing a parameter fitted to the test set, and no input label is renamed as an output. The retrieval, fact-checking, and regression steps are all applied to new QA pairs rather than to the data that produced the model weights. The self-citations (Anschütz and Groh, 2022; Ellinger, 2024) are used only as design motivation or implementation scaffolding, not as proof of the central result, and the RFVM's retrieval pipeline is not invoked as an independent theorem. The acknowledged reliance on GPT-4o and on English Wikipedia retrieval for all languages is a real limitation and a correctness risk for the generalization claim, but it is an external-knowledge coverage assumption, not a circular derivation. No step in the paper reduces, by construction or by self-citation, to its own inputs.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

Everything the central performance claim rests on beyond the supervised learning setup: defaulting to English Wikipedia as a universal knowledge source, trusting GPT-4o's prompt-executed subroutines, and accepting the shared task annotations as ground truth. The free parameters are standard hyperparameters whose values were chosen during development; none are fit to the test set, so they do not constitute circular prediction.

free parameters (5)

MMR retrieval parameters = topn=4, lambda=0.7
Chosen by trial and error (Section 3.4.1) to balance relevance and diversity of retrieved Wikipedia sentences; affects quality of evidence supplied to GPT-4o.
SVR hyperparameters = C=10, non-hallucination weight=0.01, hallucination weight=100
Set to increase sensitivity to hallucinated tokens (Section 3.4.3); these weights strongly shape the final regression output.
BERT training hyperparameters = 10+3 epochs, lr 5e-5 (BERT) / 3e-4 (FC), batch size 1, variance regularization 0.9 then 0.5
Chosen during model development (Section 3.4.2) via early stopping and validation performance.
Span merging thresholds = word distance < 3, probability difference <= 15%
Ad hoc post-processing rule (Section 3.4.3) to merge nearby tokens with similar scores; affects final span IoU.
Few-shot prompt example = one English QA pair from the Mu-SHROOM validation set
Hand-picked single example for the GPT-4o hallucination prediction prompt (Section 3.4.1); prompt quality depends on this choice.

assumptions (3)

domain assumption English Wikipedia contains the factual knowledge needed to verify answers in all 14 Mu-SHROOM languages.
The RFVM translates atomic facts to English and retrieves evidence exclusively from English Wikipedia (Section 3.1.1). If coverage for a language/culture is incomplete, token scores will be wrong. The paper's Chinese discussion acknowledges this risk.
domain assumption GPT-4o reliably carries out atomic fact extraction, translation, search-term generation, and token-level hallucination judgment.
The entire RFVM path is implemented through GPT-4o prompting (Section 3.1 and Appendix B); there is no independent verification of the extracted facts or scores beyond the final downstream metric.
domain assumption The human-annotated soft labels in Mu-SHROOM are accurate ground truth for token-level hallucinations.
The BM and SVR are trained and evaluated against these labels (Sections 3.4.2-3.4.3); any systematic annotation noise directly affects both training and reported metrics.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TUM-MiKaNi at SemEval-2025 Task 3: Towards Multilingual and Knowledge-Aware Non-factual Hallucination Identification." pith.science (2026). https://pith.science/paper/UA5IUQU6

@misc{pith2026250700579,
  author       = {Pith},
  title        = {Pith review of: TUM-MiKaNi at SemEval-2025 Task 3: Towards Multilingual and Knowledge-Aware Non-factual Hallucination Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UA5IUQU6}},
  note         = {Machine review of arXiv:2507.00579}
}

read the original abstract

Hallucinations are one of the major problems of LLMs, hindering their trustworthiness and deployment to wider use cases. However, most of the research on hallucinations focuses on English data, neglecting the multilingual nature of LLMs. This paper describes our submission to the SemEval-2025 Task-3 - Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. We propose a two-part pipeline that combines retrieval-based fact verification against Wikipedia with a BERT-based system fine-tuned to identify common hallucination patterns. Our system achieves competitive results across all languages, reaching top-10 results in eight languages, including English. Moreover, it supports multiple languages beyond the fourteen covered by the shared task. This multilingual hallucination identifier can help to improve LLM outputs and their usefulness in the future.

Figures

Figures reproduced from arXiv: 2507.00579 by the authors.

**Figure 2.** BERT-based Model architecture: BERT is prompted with the instruction, question, and answer. It is then enhanced with the answer’s part-of-speech embedding and processed through several fully connected layers to obtain the final token hallucination score. (POS) embedding. The POS embedding is generated using SpaCy (Honnibal et al., 2020), where each token in the answer is represented by a numerical POS tag. The co… view at source ↗

**Figure 3.** SVR Model architecture: POS embedding, QA entity, RFVM score, BERT annotation, BM score, and BM embedding are concatenated and processed as input for the Support Vector Regression. ing if it is part of a named entity in the question. Named entity annotations are obtained using SpaCy or Stanza (Qi et al., 2020), and the identified entities are matched to BERT tokens.This feature helps to recall non-hallucinated toke… view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Comparison between language and hard labels, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Caption 4. Precision and Completeness: - Include all relevant details from the answer. - Avoid redundancy between facts. 5. Avoid Negation: - Extract the fact as is and avoid introducing negation if the original fact does not use negation in its sentence structure. 6. …

**Figure 7.** Figure 7: Retrieval-based Model architecture [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 27 canonical work pages

[1]

he," "she,

Coreference Resolution: - Resolve pronouns (e.g., "he," "she," "it") to their specific referents. - Resolve demonstratives (e.g., "this," "that") to their explicit meaning

work page
[2]

In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6528–6548, Miami, Florida, USA

An audit on the perspectives and challenges of hallucinations in NLP. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6528–6548, Miami, Florida, USA. Association for Computational Linguistics. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural langu...

arXiv 2024
[3]

- Maintain semantic accuracy and avoid splitting at inappropriate junctures (e.g., splitting compound phrases unnecessarily)

Logical Breakdown: - Split the information into the smallest meaningful units. - Maintain semantic accuracy and avoid splitting at inappropriate junctures (e.g., splitting compound phrases unnecessarily). AR CA CS DE EN ES EU FA FI FR HI IT SV ZH0 50 100 0.2 0.4 0.6 AR CA CS DE EN ES EU FA FI FR HI IT SV ZH0 50 100 0.2 0.4 0.6 AR CA CS DE EN ES EU FA FI F...

work page
[4]

- Avoid redundancy between facts

Precision and Completeness: - Include all relevant details from the answer. - Avoid redundancy between facts

work page
[5]

Contextual Dependency: - Ensure each fact is self-contained and does not rely on the context of the question or other facts

work page
[6]

english_translation,

Language Handling: - The input question and answer can be in a language other than English. - Provide the extracted fact in its original language. - Add an additional key, "english_translation," containing the English translation of the fact for each atomic fact

work page
[7]

### Task Prioritization:

Formatting: - Ensure the output is a valid JSON list. ### Task Prioritization:

work page
[8]

Avoid Negation: - Extract the fact as is and avoid introducing negation if the original fact does not use negation in its sentence structure

work page

Show all 29 references

[9]

**Handle Ambiguity Carefully**: If a term could refer to multiple topics, include context or disambiguation when necessary
[10]

**Align with Wikipedia Titles**: Generate terms that match Wikipedia article titles or redirects
[11]

Prioritize accuracy over brevity
[12]

Always focus on breaking down complex information into the most granular, standalone truths while maintaining the semantic integrity of the original answer

Ensure all facts are clear, unambiguous, and self-contained. Always focus on breaking down complex information into the most granular, standalone truths while maintaining the semantic integrity of the original answer. B.2 Search Term Generation System Prompt You are an assista...
[13]

sentence

**Key "sentence"**: The original fact from the input
[14]

search_terms

**Key "search_terms"**: A list of concise search terms (strings) that are relevant and likely to lead to Wikipedia pages. ### Guidelines for Generating Search Terms:
[15]

David Sandburg

**Use the Question Context**: Integrate the question to determine the main concepts, correct spelling of entities, and relevance. Use it to verify or correct typos in names (e.g., if "David Sandburg" appears, check if it should be "David Sandberg")
[16]

**Focus on the Core Concepts**: Prioritize search terms that match the question 's main concept and facts, ensuring relevance to the broader QA context
[17]

2008 Summer Olympics

**Retain Exact Meanings**: Preserve the exact meaning of extracted concepts. For example, if "2008 Summer Olympics" appears, keep "2008 Summer Olympics" and not just "Summer Olympics" to ensure specificity

2008
[18]

**Incorporate Atomic Facts**: Use the facts to refine search terms, but always keep them anchored to the core idea of the question-facts pair
[19]

Avoid unnecessary qualifiers or verbose phrases

**Keep It Concise**: Use the shortest terms that retain relevance. Avoid unnecessary qualifiers or verbose phrases
[20]

**Balance Specificity and Relevance**: Avoid terms that are too broad or too detailed to match Wikipedia pages
[21]

**Exclude Irrelevant Information**: Ignore filler words, minor details, or auxiliary information that does not contribute to the main concept
[22]

Use subwords only if they represent a distinct concept

**Avoid Over-Specific Subterms**: Do not fragment terms excessively. Use subwords only if they represent a distinct concept
[25]

**Abstract or General Statements**: For facts without clear entities, infer general topics while aligning with the question and factual context
[26]

**Provide At Least One Term**: Ensure each fact has at least one concise search term, unless it is too abstract to generate one
[27]

question

**Return a Properly Formatted JSON String**: - Ensure the output is valid JSON. - Correctly escape characters. - Avoid trailing commas or mismatched brackets. - Format the output to be compact. ### Input Format: A JSON object with these keys: - **"question"**: A string represe...

1905
[28]

Review the question, the answer, and the provided facts to determine whether the token is likely correct or incorrect

**Reasoning**: First, analyze each token internally. Review the question, the answer, and the provided facts to determine whether the token is likely correct or incorrect. Evaluate the relevance of the `page_title` and its corresponding facts before using them to verify the an...
[29]

Stoveren

**Conclusion**: After reasoning, output your final classifications in the required JSON structure, ensuring the classification for each token appears last, after reasoning is complete. ### Handling Typos: - Identify typos by comparing tokens in the answer and question with nam...

1972
[2024]

In Proceedings of the 18th Interna- tional Workshop on Semantic Evaluation (SemEval- 2024), pages 1788–1797, Mexico City, Mexico

HIT-MI&T lab at SemEval-2024 task 6: DeBERTa-based entailment model is a reliable hallu- cination detector. In Proceedings of the 18th Interna- tional Workshop on Semantic Evaluation (SemEval- 2024), pages 1788–1797, Mexico City, Mexico. As- sociation for Computational Linguis...

2024
[2025]

question

How much do llms hallucinate across lan- guages? on multilingual estimation of llm hallucina- tion in the wild. Preprint, arXiv:2502.12769. Raúl Vázquez, Timothee Mickus, Elaine Zosa, Teemu Vahtola, Jörg Tiedemann, Aman Sinha, Vincent Segonne, Fernando Sánchez-Vega, Alessandro...

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

he," "she,

Coreference Resolution: - Resolve pronouns (e.g., "he," "she," "it") to their specific referents. - Resolve demonstratives (e.g., "this," "that") to their explicit meaning

work page

[2] [2]

In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6528–6548, Miami, Florida, USA

An audit on the perspectives and challenges of hallucinations in NLP. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6528–6548, Miami, Florida, USA. Association for Computational Linguistics. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural langu...

arXiv 2024

[3] [3]

- Maintain semantic accuracy and avoid splitting at inappropriate junctures (e.g., splitting compound phrases unnecessarily)

Logical Breakdown: - Split the information into the smallest meaningful units. - Maintain semantic accuracy and avoid splitting at inappropriate junctures (e.g., splitting compound phrases unnecessarily). AR CA CS DE EN ES EU FA FI FR HI IT SV ZH0 50 100 0.2 0.4 0.6 AR CA CS DE EN ES EU FA FI FR HI IT SV ZH0 50 100 0.2 0.4 0.6 AR CA CS DE EN ES EU FA FI F...

work page

[4] [4]

- Avoid redundancy between facts

Precision and Completeness: - Include all relevant details from the answer. - Avoid redundancy between facts

work page

[5] [5]

Contextual Dependency: - Ensure each fact is self-contained and does not rely on the context of the question or other facts

work page

[6] [6]

english_translation,

Language Handling: - The input question and answer can be in a language other than English. - Provide the extracted fact in its original language. - Add an additional key, "english_translation," containing the English translation of the fact for each atomic fact

work page

[7] [7]

### Task Prioritization:

Formatting: - Ensure the output is a valid JSON list. ### Task Prioritization:

work page

[8] [8]

Avoid Negation: - Extract the fact as is and avoid introducing negation if the original fact does not use negation in its sentence structure

work page