REVIEW 3 major objections 7 minor
CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?
T0 review · 3 major / 7 minor · reviewed 2026-07-11 · grok-4.5
Pith's one-line read Internal hallucination signals extracted from LLM activations transfer across languages and domains for most models.
desk verdict First clean transfer study of internal-state hallucination detectors across English/Arabic and two Arabic domains; results hold for most of six models once you accept the GPT-4o labels. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
CrossHallu feature vectors: Wasserstein distances and cosine similarities between consecutive even/odd layers of hidden states and attention, plus eight token-probability statistics (min-of-max, max spread, normalized entropy, low-probability counts, mean gradient, and three percentiles), concatenated and fed to a three-layer MLP binary detector.
What would settle it
Replace the GPT-4o labels with a large human-verified set (or a second independent judge that agrees strongly with humans) for both languages and re-run every transfer experiment; if cross-lingual AUC-ROC collapses while monolingual scores stay high, the transfer claim is false.
Extended reading notes
Core claim
Internal-state hallucination signals from hidden states, attention patterns, and token probabilities transfer across languages and domains for most of the six evaluated LLMs. Cross-lingual success is governed by class separability within each language and by how well English and Arabic embeddings overlap; cross-domain transfer within Arabic is feasible yet asymmetric, with training on TruthfulQA Arabic transferring better to HalluScore than the reverse.
Load-bearing premise
The GPT-4o binary labels for English and Arabic answers are accurate enough that measured transfer reflects real internal-signal generalization rather than shared judge artifacts.
Editorial extensions
If this is right
- A detector trained only on English internal states can be applied to Arabic generations for models whose English and Arabic embeddings already interleave.
- Training on a cleaner Arabic QA set can raise detection on a harder Arabic set, so data quality matters more than matching the exact target domain.
- Multilingual models with shared language-agnostic subspaces are better zero-shot monitoring candidates than Arabic-centric models.
- Combined language-plus-domain shift remains the hardest case, so production systems may still need light target calibration.
- The same internal features can diagnose how language-agnostic a model’s deeper layers actually are.
Reading between the lines
- The same transfer protocol could probe whether other reliability signals (toxicity, bias) also live in shared subspaces without new labeled data.
- If late-layer cosine drops are the main discriminative cue, detectors could be restricted to upper layers only, cutting extraction cost.
- Extending the pair set beyond Arabic–English would likely show sharper transfer collapse once morphological or script distance breaks feature-space alignment.
- Asymmetric domain transfer implies that easier, cleaner hallucination corpora may be more valuable as source data than large noisy ones.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. CrossHallu is an empirical study of whether internal-state features used for hallucination detection (hidden-state and attention Wasserstein/cosine shifts plus token-probability statistics, following HalluShift) transfer across languages and domains. The authors evaluate six Arabic-capable LLMs on generative QA under monolingual baselines, English↔Arabic transfer on aligned TruthfulQA, Arabic cross-domain transfer (TruthfulQA-AR ↔ HalluScore), and combined language+domain transfer, reporting F1, AUC-ROC, and PR-AUC for a fixed three-layer MLP. They conclude that internal hallucination signals transfer for most models, that cross-lingual success tracks class separability and English–Arabic feature-space alignment (supported by t-SNE and layer-wise cosine trajectories), and that Arabic cross-domain transfer is asymmetric (Ar→HS often better than HS→Ar).
Significance. The paper fills a clear gap: internal-state hallucination detectors have been studied almost exclusively in English, while multilingual deployment and higher hallucination rates in lower-resource languages make transferability practically important. The experimental matrix cleanly isolates language shift, domain shift, and their combination; the public code, multi-model coverage, and supporting t-SNE / layer-wise analyses are genuine strengths. If the transfer patterns hold under more carefully validated labels and stricter normalization, the work would provide useful guidance on which model families support language-agnostic hallucination probes and on when training on a cleaner source domain helps a harder target. The contribution is primarily empirical and methodological rather than theoretical.
major comments (3)
- §4.4 Feature normalization: for cross-lingual (and by implication CL-CD) runs, a QuantileTransformer is “fit independently on the training and test feature sets.” Using test-set quantiles adapts the target representation with target statistics and is not pure zero-shot transfer of a detector trained only on the source. This choice is load-bearing for the central transfer claim in Table 1 (CL and CL-CD blocks). Please either (i) report the main numbers with a scaler fit only on the source/train features and applied to the target, or (ii) reframe those settings as unsupervised feature alignment / domain adaptation and add the pure-transfer numbers as the primary result. The monolingual StandardScaler protocol is fine.
- §3.2 and Appendix D: every F1/AUC number in Table 1 is defined against binary labels from a single GPT-4o judge (temperature 0, four rules, bilingual CoT prompts). There is no human agreement, multi-judge consensus, or even GPT-4o–BLEURT/human correlation on the Arabic subset, despite the paper’s own note that BLEURT is weak on Arabic and that TruthfulQA/HalluScore contain misconception and culturally nuanced items. This is the softest load-bearing premise for the transfer conclusions. At minimum, provide human validation (or multi-judge agreement) on a stratified subset of English and Arabic labels, report agreement statistics, and discuss residual language- or culture-dependent judge bias as a threat to interpreting “signal transfer.”
- Table 1 and §5 Cross-domain: HalluScore is introduced as “currently the only available Arabic dataset specifically designed for hallucination detection” and is authored by two of the present authors (Alansari & Luqman, 2026). Using predefined splits is appropriate, but the Ar→HS gains (e.g., Phi4-mini 57.19→72.59, Aya 80.37→86.90) and the asymmetry argument rest on treating HalluScore as an independent harder domain. Please add an explicit discussion of possible construction/overlap effects between TruthfulQA-style misconception items and HalluScore, and, if feasible, a small external Arabic QA set or a leave-one-category-out check so that cross-domain claims are not solely internal to the authors’ own benchmark.
minor comments (7)
- Table 1 header and caption: “1st, 2nd, 3rd best AUC-ROC per row” is stated but the manuscript text does not specify how ties or near-ties are broken; a short note would help.
- §3.1: the token-probability feature list mentions “max/min entropy, low-prob count, gradients, percentiles” in Figure 2 but the prose derives eight scalars from pmax/pmin; a single explicit feature-dimension table (per model if dimensions differ) would improve reproducibility.
- Figure 4 / Figure 6: layer-pair indices differ across models (different depths); stating L and the exact even/odd index sets per model in the caption or appendix would reduce ambiguity.
- §4.1: 772/817 TruthfulQA items retained after translation; a brief characterization of excluded items (categories, failure modes) would help assess selection bias in the bilingual subset.
- PR-AUC is repeatedly high due to class imbalance (§5); consider reporting positive-class prevalence per model/dataset in Table 1 or Appendix E so readers can calibrate F1/PR-AUC.
- Acknowledgments section still contains the ACL template boilerplate (“This document has been adapted by Steven Bethard…”); replace with actual acknowledgments or remove.
- Code link in the abstract points to github.com/aishaalansari57/CrossHal while Appendix C cites an anonymous.4open URL; align the public repository reference for the camera-ready version.
Circularity Check
No significant circularity; purely empirical transfer measurement with standard train/test splits and external baselines.
full rationale
The paper's central claims are empirical measurements of MLP detector performance (F1/AUC-ROC/PR-AUC) under monolingual, cross-lingual, cross-domain, and combined transfer settings on features extracted from six LLMs. Feature extraction follows the external HalluShift pipeline (hidden-state/attention Wasserstein+cosine, token-probability statistics); labels come from an independent GPT-4o judge under explicit rules; the detector is a standard three-layer MLP trained with BCE/Adam on fixed 75/25 or predefined splits. No equation, parameter, or decision boundary is defined in terms of the quantity later reported as a 'prediction' or 'transfer result.' HalluScore (co-authored by two of the present authors) is used only as one evaluation set with its published train/test split; it does not define the detector, the features, or the transfer metric. Self-citations to the authors' prior surveys and the HalluScore paper supply background and a dataset, not a uniqueness theorem or ansatz that forces the reported numbers. Once the (externally generated) labels are taken as given, every reported transfer score is an ordinary out-of-sample evaluation and does not reduce to its inputs by construction. Score 0 is therefore the correct assessment.
Assumptions & free parameters
free parameters (4)
- MLP hidden sizes / dropout / lr / epochs
- low-probability threshold τ
- layer-subsampling stride
- QuantileTransformer vs StandardScaler choice
assumptions (3)
- domain assumption Distributional shifts in consecutive-layer hidden states, attention tensors, and token-probability statistics are informative of hallucination (inherited from HalluShift).
- domain assumption GPT-4o with the supplied bilingual CoT prompt produces labels that are sufficiently accurate for both English and Arabic to serve as ground truth.
- domain assumption The 772 retained TruthfulQA items after manual translation filtering remain distributionally comparable to the original English set.
Cite this review
Pith. "Pith review of CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?." pith.science (2026). https://pith.science/paper/2Q66Y3Q5
@misc{pith2026260704029,
author = {Pith},
title = {Pith review of: CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?},
year = {2026},
howpublished = {\url{https://pith.science/paper/2Q66Y3Q5}},
note = {Machine review of arXiv:2607.04029}
}
read the original abstract
Recent hallucination detection techniques in large language models (LLMs) focus on directly extracting features from a model's internal representations and training a classifier on these features to detect hallucinations, demonstrating promising results. Notwithstanding this advancement, most internal-state hallucination detection techniques have been explored predominantly in English, raising the question of whether such internal signals generalize across different languages and domains. To address this gap, we present CrossHallu, the first study to evaluate the cross-lingual and cross-domain generalization of hallucination detection using internal representations from six LLMs on the generative question-answering task. We conduct a systematic Arabic <-> English evaluation using TruthfulQA, an Arabic translated version of TruthfulQA, and HalluScore. This evaluation encompasses monolingual training and testing, cross-lingual transfer, cross-domain transfer, and combined cross-lingual and cross-domain transfer. The results reveal that internal-state hallucination signals in LLMs transfer across languages and domains for most models, with cross-lingual performance highly dependent on both class separability and language alignment in the feature space, whereas cross-domain transfer within Arabic varies depending on the training and testing datasets used for the hallucination detector. The code is publicly available at https://github.com/aishaalansari57/CrossHal.
Figures
Figures from the paper (7 more)
Reviewed July 11, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.