{"id":"a4eaf77c-6575-4160-b7c4-1556ff02c015","arxiv_id":"2506.03357","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multilingual hallucination detector that flags words where a language-specialized model's perplexity diverges from the rest, achieving IoU around 0.3.","lead":"Ask a Local detects hallucinated text in AI answers by comparing how surprised language-specialized models are. It works across 14 languages without training, but its accuracy is modest and a key formula appears undefined as written.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 1 defines H as KL between two scalar perplexity values, but KL divergence is undefined for scalars; as written the central formula is not computable, so the reported IoU/Spearman results cannot be tied to the claimed mechanism without clarification or code inspection.","rationale":"The paper's strongest claim is that the method achieves IoU around 0.3 and comparable Spearman values on a 14-language human-annotated dataset without external data or training. For that claim to hold, the score in Equation 1 must be well-defined and implementable. It is not, because KL divergence requires two distributions, while PPL_local and PPL_foreign are scalars. This is the same load-bearing concern the reader identified, and it is the most direct obstacle to accepting the results: no amount of empirical success can compensate for an undefined central formula unless the authors clarify the intended operation. The reader's verdict of CONDITIONAL is appropriate, and I agree with it. I considered whether a different issue, such as the lack of a baseline showing that divergence beats average perplexity, is more serious; that is a real concern, but it depends on first making Equation 1 computable. I also considered the modest IoU values, but the paper openly compares against a stronger published method (around 0.6) and frames its contribution as a training-free multilingual approach, so low absolute performance is not a fatal flaw. The paper's honest limitations section, the use of a public human-annotated dataset, and the release of code are positive features. However, the undefined Equation 1 is a correctness risk that must be resolved before the central mechanism can be evaluated. A concrete test is to check the released code's implementation and to rerun the evaluation with explicit scalar divergence candidates; that will settle whether the issue is merely a notational error or a substantive gap.","tokens_in":7852,"tokens_out":3623,"duration_ms":43605,"concrete_test":"Inspect the released repository (github.com/ACMCMC/ask-a-local) and locate the exact implementation corresponding to Equation 1; identify what scalar operation, if any, is computed between PPL_local and PPL_foreign. Then re-run the English validation-set evaluation with three well-defined scalar alternatives (x*log(x/y), (x-y)^2, |x-y|) and compare IoU and Spearman correlation. If the implemented operation differs from KL, or if the metrics change materially across the three alternatives, the paper's central formula must be corrected and the robustness of the headline result re-evaluated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central hallucination score in Equation 1 is H(w_i) = beta * KL(PPL_local(w_i) || PPL_foreign(w_i)) + (1-beta) * average perplexity. However, Equations 2 and 3 define PPL_local(w_i) and PPL_foreign(w_i) as scalar weighted sums of model perplexities for a single word. KL divergence is a functional over probability distributions, not a distance or difference between two scalars. Therefore, Equation 1 is mathematically undefined as written, and the method cannot be reproduced from the manuscript. This is not a cosmetic typo: the tuned beta is 0.496, so roughly half of the score's weight is assigned to an unspecified operation, and the paper's title and contribution rest on 'model divergence.' If the released code actually implements a different operation (for example, a log-ratio, squared difference, or a KL over a distribution of words), then the paper must state that formula; otherwise, Table 3's results cannot be attributed to the stated divergence mechanism. A secondary but related gap is that no ablation isolates the divergence term: metrics for beta=0 (average perplexity only) and beta=1 (divergence only) are not reported, so there is currently no evidence that the specialized-divergence component adds signal beyond generic perplexity. The undefined KL is the more load-bearing issue because it blocks even a correct re-implementation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes “Ask a Local”, a training-free hallucination detection method that compares perplexities of language-specialized models to identify hallucinated spans in multilingual LLM outputs. The method computes per-word perplexities with several specialized models, normalizes them using question tokens, and combines a weighted “local” versus “foreign” perplexity divergence with an average-perplexity term into a single score H(w_i). Hyperparameters are tuned on the English validation split, and the final method is evaluated on a 14-language human-annotated question-answer dataset, reporting IoU scores around 0.3 and Spearman correlations around 0.2–0.4. The authors argue that the approach generalizes across languages without external data or training and release code publicly.","tokens_in":8162,"tokens_out":3704,"duration_ms":42624,"significance":"If the method works as described, the paper offers a useful and reproducible contribution: it is training-free, does not require external knowledge sources, and explicitly targets multilingual hallucination detection, with code released and hyperparameters tuned on a separate validation set. The core intuition—that a domain-specialized model will be more surprised by errors in its domain—is attractive and worth testing. However, the central score equation is mathematically undefined as written, and the experiments do not isolate the divergence component from generic perplexity. These issues currently prevent the reported numbers from being attributed to the claimed mechanism. With a corrected formulation and a proper ablation, the contribution could be credible, but in its present state the central claim is not reproducible.","major_comments":[{"comment":"The hallucination score H(w_i) is defined as beta * KL(PPL_local(w_i) || PPL_foreign(w_i)) + (1-beta) * average perplexity, but PPL_local and PPL_foreign are scalars—weighted sums of scalar word-level perplexities in Eqs. (2) and (3). The Kullback-Leibler divergence is defined for probability distributions, not for two scalars. Therefore Eq. (1) is not computable as written. Since the tuned beta is 0.496, roughly half of the score is assigned to an undefined operation. The manuscript must either state the correct divergence formula (for example, a KL over a distribution of token or word perplexities, a log-ratio, or a squared difference) or point to released code that unambiguously defines the operation; otherwise the IoU and Spearman results in Table 3 cannot be tied to the stated divergence mechanism.","section":"Section 2.1, Eq. (1)"},{"comment":"No ablation isolates the divergence term. Setting beta=0 in Eq. (1) reduces the score to average perplexity alone, and setting beta=1 reduces it to the divergence alone. Reporting these two extremes, along with the tuned intermediate value, on the validation or test sets would show whether specialized-model divergence adds signal beyond generic perplexity. Without such an ablation, the paper's central claim—that divergence between specialized models drives hallucination detection—is not supported by the reported experiments.","section":"Section 3.1 and Section 4"},{"comment":"The baselines are too weak to support the performance claims. Comparing against a “neural baseline (close to 0)” and “marking all words as hallucinated” does not position the method relative to the state of the art, even though the manuscript itself cites Vázquez et al. (2025) as achieving scores around 0.6 on similar metrics. A comparison on the same dataset against existing / top-performing systems is necessary to substantiate the claim of “comparable Spearman correlation values” and the broader cross-lingual effectiveness of the method.","section":"Section 4"}],"minor_comments":[{"comment":"The heading “Addessing differences in tokenization” contains a typo; it should read “Addressing differences in tokenization.”","section":"Section 2.2 heading"},{"comment":"Equation (3) has a stray “nX i” subscript after the summation symbol, which appears to be a leftover from LaTeX source; it should be removed or corrected.","section":"Section 2.1, Eq. (3)"},{"comment":"The text says the instruction-following model outputs probabilities over model-name tokens and that these probabilities are used to assign weights, but the subsequent softmax formula applies to logits divided by temperature. Please clarify whether the softmax is computed over logits or over token probabilities; these are different quantities.","section":"Section 2.4"},{"comment":"The table reports a single scalar per language without confidence intervals or significance tests, despite only 200 examples per language; adding such information would strengthen the cross-lingual generalization claim.","section":"Section 4, Table 3"},{"comment":"The Limitations section states that “our results are heavily influenced by the structure of these human annotations,” which is an important caveat but should also be reflected in the abstract and conclusion where the results are presented as consistent cross-lingual performance.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly positioned as an early-stage method paper, but the undefined KL in Eq. (1) is a serious correctness issue that must be resolved before the results can be trusted. I recommend that the editor ensure the authors either reformulate the score in full mathematical detail or verify that the released code implements a well-defined operation, and that the revised version include an ablation of the divergence term plus a comparison with actual state-of-the-art baselines on the same dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper has a genuinely useful idea — zero-training multilingual hallucination detection via divergence between language-specialized models — but as written the central formula is undefined, and that blocks reproduction. Worth sending to review, not worth accepting as is.\n\nWhat's actually new: the combination of a weighted ensemble of Goldfish models and a general model, with weights assigned by an instruction-following model that picks the most relevant specialist for a question-answer pair, then using the divergence between local and foreign perplexities to flag hallucinated spans. That's a real extension of white-box perplexity methods, and the 14-language evaluation on human-annotated data is a solid contribution. The handling of tokenization mismatches and perplexity scale differences is thoughtful, and the limitations section is candid about annotation noise and language-country mismatches.\n\nThe soft spot is load-bearing: Equation 1 defines H as beta times KL(PPL_local || PPL_foreign) plus an average perplexity term. But Equations 2 and 3 define PPL_local and PPL_foreign as scalar weighted sums over models. KL divergence is a functional on probability distributions, not a distance between two scalars. As written, the formula is not computable. This is not a typo — half the score weight (beta=0.496) comes from this term. The authors release code, so it's possible they implement something else, but the manuscript must say what that something else is. Without that, the reported IoU/Spearman numbers can't be tied to the stated mechanism.\n\nA related gap: no ablation isolates the divergence term. Beta=0 (average perplexity only) and beta=1 (divergence only) are not reported, so there's no evidence the divergence adds anything beyond generic perplexity.\n\nThe baseline comparison is also weak. They mention other methods reaching ~0.6 IoU but don't run them on this dataset. And hyperparameters tuned on English validation only might hide language-specific behavior, though that's a minor issue given compute.\n\nWho this is for: researchers working on hallucination detection or multilingual evaluation. The idea is a legitimate heuristic worth discussing, and the dataset/annotation discussion is useful. But the paper needs a major revision: fix the divergence definition, add proper baselines, and run the ablation.\n\nYes, send to peer review — a serious referee could push for those fixes.","headline":"A promising zero-training multilingual hallucination detector, but the central KL formula as written is undefined and blocks reproduction.","tokens_in":8584,"tokens_out":3405,"would_cite":false,"duration_ms":37157,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Comparing language-specialized models' perplexities locates hallucinated spans in 14 languages without training or external data.","keywords":["hallucination detection","multilingual","perplexity divergence","specialized language models","zero-shot detection","span tagging","large language models","factual correctness"],"falsifier":"Re-running the 14-language evaluation with the divergence term replaced by a straightforward absolute difference $|\\mathrm{PPL}_{\\text{local}} - \\mathrm{PPL}_{\\text{foreign}}|$ — which is defined for scalars — would show whether the reported IoU of about 0.3 depends on the particular formula or on the domain-surprise intuition itself.","tokens_in":7702,"feed_emoji":"🌐","tokens_out":8226,"duration_ms":85214,"temperature":0.7,"pith_summary":"This paper introduces a hallucination detector that needs no training data and no external database, only a handful of language-specialized language models. The idea is that a model trained mainly on, say, Spanish should be more puzzled by a factual error about Spain than a model that knows nothing special about Spain; the gap in their per-word perplexities marks the error. The method averages each word's perplexity under specialized models into 'local' and 'foreign' numbers, takes a divergence between them, and turns high-scoring words into hallucinated spans with the help of an instruction-following model. On a human-annotated dataset in 14 languages, the detector scores Intersection-over-Union around 0.3, with Italian and Catalan reaching 0.42 and 0.38. The authors take this as evidence that the divergence signal transfers across languages without adaptation.","feed_headline":"Model perplexity gaps expose hallucinations in 14 languages","feed_subtitle":"Zero-training method scores about 0.3 IoU on human-annotated answers, with Italian and Catalan highest.","key_machinery":"The load-bearing object is the per-word hallucination score $H(w_i)=\\beta\\, \\mathrm{KL}(\\mathrm{PPL}_{\\text{local}}(w_i)\\,\\|\\,\\mathrm{PPL}_{\\text{foreign}}(w_i))+(1-\\beta)\\frac{1}{|M|}\\sum_j \\mathrm{PPL}(w_i;m_j)$, with $\\mathrm{PPL}_{\\text{local}}$ and $\\mathrm{PPL}_{\\text{foreign}}$ defined as weighted sums over specialized models. The weights $\\alpha_j$ come from an instruction-following model asked to pick the most relevant specialization for the question-answer pair, with a temperature-smoothed softmax over the first token of each model's name. Perplexities are normalized to zero mean and unit variance using question tokens, then aggregated per word by taking the maximum of the word's token perplexities. A threshold $\\sigma$ selects candidate words and the same instruction-following model proposes span boundaries; overlapping span probabilities are averaged. The specialized models are the Goldfish family for languages, with a general-knowledge model and a math model as optional extras, and an instruction-following model to assign weights and tag spans.","core_discovery":"The central discovery is that divergence between the perplexities of language-specialized models is a workable cross-lingual signal for hallucinated spans. Concretely, for each word $w_i$ the method builds $\\mathrm{PPL}_{\\text{local}}(w_i)$ as an $\\alpha$-weighted average of perplexities from models specialized in the question's language/domain and $\\mathrm{PPL}_{\\text{foreign}}(w_i)$ as the complementary weighted average; the hallucination score is $\\beta$ times the KL divergence between these two aggregates plus $(1-\\beta)$ times the average perplexity. After normalizing perplexities using question tokens, words whose score exceeds $\\sigma$ standard deviations above the answer mean are selected, and an instruction-following model converts them into complete spans. Tested on a human-annotated 14-language dataset, the method obtains IoU around 0.3 and comparable Spearman correlations, strongest on Italian (0.42) and Catalan (0.38), with no language-specific tuning.","pith_inferences":["The same divergence principle could transfer to non-linguistic specializations (code, medicine, law), where 'local' models would be domain experts rather than language experts.","The undefined KL on scalars suggests the numerical form in Equation 1 is likely a placeholder; a proper divergence defined over the distribution of perplexities across words might yield a more stable score.","The reliance on an instruction-following model for span proposals means the method is not fully unsupervised; an entirely automatic span-extraction heuristic would strengthen the zero-training claim.","The dependency on per-language specialized models implies the method's ceiling is bounded by model coverage and quality; for low-resource languages without such models, the signal weakens."],"forward_implications":["The detector needs no labeled hallucination data and no retrieval or external knowledge base, so it can be applied immediately to any language for which a specialized model exists.","Because performance stays near 0.3 IoU across all 14 languages without adaptation, the divergence signal appears language-agnostic, a property most trained detectors lack.","The best scores on Italian and Catalan suggest that the quality or number of specialized models per language directly affects detection, pointing to model coverage as a lever.","The span-tagging step produces complete meaning units (entities, clauses) rather than isolated words, matching the granularity human annotators use.","Using computationally light specialized models keeps the pipeline scalable to many languages; the extra general and math models are optional."],"supporting_citations":[{"why":"Provides the human-annotated 14-language question-answer dataset used for validation and testing, and the comparative scores the method is measured against.","marker":"Vázquez et al. (2025)"},{"why":"Supplies the Goldfish language-specialized models whose perplexity divergence is the core signal.","marker":"Chang et al. (2024)"},{"why":"Provides the instruction-following model used to assign specialization weights and to propose hallucinated span boundaries.","marker":"AI@Meta (2024)"}],"fun_headline_variants":["Zero-training perplexity divergence finds LLM hallucinations across 14 languages","Specialized model surprise exposes hallucinated spans in any language","Ask a Local: multilingual hallucination detection without fine-tuning","Model divergence flags false facts in 14 languages, no training needed","Perplexity gaps between local experts reveal LLM fabrications"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a specialized model is more surprised by errors in its domain, and that the paper's 'divergence' between two scalar perplexity values (written as a Kullback-Leibler divergence, though KL is not defined for scalars) detects that surprise.","fun_headline_variants_meta":{"raw":{"variants":["Zero-training perplexity divergence finds LLM hallucinations across 14 languages","Specialized model surprise exposes hallucinated spans in any language","Ask a Local: multilingual hallucination detection without fine-tuning","Model divergence flags false facts in 14 languages, no training needed","Perplexity gaps between local experts reveal LLM fabrications"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3847,"prompt_tokens":969,"completion_tokens":2878,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2792}},"tokens_in":585,"tokens_out":2878,"duration_ms":24526,"temperature":1.0,"reasoning_tokens":2792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:04:14.357564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the 14-language evaluation with the divergence term replaced by a straightforward absolute difference $|\\mathrm{PPL}_{\\text{local}} - \\mathrm{PPL}_{\\text{foreign}}|$ — which is defined for scalars — would show whether the reported IoU of about 0.3 depends on the particular formula or on the domain-surprise intuition itself.","supporting_citations":[],"review_version":1}