{"id":"d21a2faf-8ca8-4bb1-abbe-c326d208a808","arxiv_id":"2504.18639","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A SemEval-2025 system combines semantic role labeling, textual entailment, and logit confidence to detect hallucinated spans, but its confidence formula degenerates to one over unit length.","lead":"A shared-task system paper describes a pipeline that detects hallucinated spans by splitting answers into semantic roles, checking them against a GPT-4-retrieved context, and blending entailment scores with token logits. The claim that logit confidence improves detection is undermined by an equation that makes the confidence term a constant.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (1)'s logit confidence score is mathematically equal to 1/n for every unit, so the claimed token-level confidence refinement is vacuous and the combined-score central mechanism is unsupported.","rationale":"The reader's weakest_assumption identifies the same load-bearing flaw: Eq. (1) collapses to 1/n. My reading confirms it exactly from the manuscript's own displayed example. There is no need to manufacture a different objection; this one is sufficient to reject the central claim that entailment scores are refined by genuine token-level confidence. The lack of baselines, error bars, code, and alpha values compounds the problem but is secondary. I therefore agree with the reader's REJECT verdict and recommend no change to it. Independent credit is minimal: the paper provides no machine-checked proofs or reproducible code, and no comparison to an entailment-only version.","tokens_in":6674,"tokens_out":4977,"duration_ms":47972,"concrete_test":"Take any two 3-token and 9-token units with arbitrary output logits and apply Eq. (1) with the denominator restricted to the unit's tokens; the scores will always be exactly 1/3 and 1/9, respectively. Then rerun the English and Arabic pipelines after replacing logit_score with the constant 1/n, using the alpha that is currently undisclosed. If IoU/Cor do not change, the confidence component is vacuous as argued; if they do change materially, the authors must supply the alpha, the softmax denominator, and token-count statistics to demonstrate that any confidence signal beyond length is present.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 defines Eq. (1) as logit_score = (1/n) \\sum_i e^{logit_i} / \\sum_j e^{logit_j}. With the denominator summed over the n tokens of the same atomic unit, the bracketed terms sum to 1 for any logits, forcing logit_score = 1/n. The manuscript's own examples confirm this: a 3-token unit ('a silver medal') receives 0.3333333333333333 and a 9-token unit receives 0.1111111111111111, values that depend only on token count. Thus Eq. (2) reduces to refined_score = alpha * entailment + (1-alpha) / n, so the confidence term carries no model-uncertainty information. Because the abstract and Section 4.5 credit this confidence term with refining entailment scores, the reported IoU values (0.358 English, 0.28 Arabic) do not support the paper's central claim. The absence of an alpha value or baseline makes the reported numbers independently unreproducible; even the worked example points to alpha ~ 0.6, which is never disclosed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a span-level hallucination detection system for the SemEval-2025 Mu-SHROOM shared task. The method decomposes LLM-generated answers into semantic roles via Semantic Role Labeling, retrieves a reference context through GPT-4 prompting, scores each role with a DeBERTa-based textual entailment model, and combines the entailment score with a token-level logit-based confidence score (Eq. 1) into a refined score (Eq. 2). Units with refined score below 0.5 are marked as hallucinated. Experiments on English and Arabic report IoU 0.358/Cor 0.322 and IoU 0.28/Cor 0.21 respectively, with additional GPT-4/LLaMA verification of detected spans.","tokens_in":6908,"tokens_out":7813,"duration_ms":69877,"significance":"If the proposed confidence refinement were valid, the paper would provide a practical, language-aware approach to span-level hallucination detection that combines semantic decomposition with model uncertainty. The SRL-based decomposition and the comparison of two Arabic parsing pipelines are useful engineering contributions. However, the central confidence mechanism is mathematically degenerate, no baseline or ablation is reported, and the evaluation has a self-referential component. As a result, the scientific claims are not currently supported; the paper is better viewed as a system description that needs substantial correction.","major_comments":[{"comment":"The confidence score is degenerate. Because the softmax denominator sums over the same n tokens as the numerator, the average of the n probabilities is identically 1/n regardless of the logit values. The paper's own examples confirm this: 'a silver medal' yields logit_score 0.3333 and the 9-token 'in the 2008 Summer Olympics in Beijing , China' yields 0.1111, exactly 1/n. Consequently Eq. (2) reduces to refined_score = α·entailment + (1−α)/n, and the confidence component carries no information about token-level uncertainty. The abstract and §4.5 credit this term with refining the entailment scores, but that claim is unsupported by the equation as written. The authors must correct the normalization (e.g., define the denominator over the model's output vocabulary or over a fixed token set) or remove the confidence component entirely and re-evaluate the system.","section":"§4.3, Eq. (1)"},{"comment":"The evaluation is not reproducible and does not support the claimed contribution. The hyperparameter α and the procedure for setting the refined-score threshold of 0.5 are not reported; the worked example in §5.2 implicitly suggests α ≈ 0.6 (if the entailment probability is taken as 0.007), but this is never stated. There is no ablation that removes the confidence term, no comparison to an entailment-only baseline or to other Mu-SHROOM systems, and no error bars or significance tests. Given that Eq. (1) is degenerate, an ablation is essential to determine whether the integrated score outperforms the entailment score alone. Without these, the reported IoU/Cor values in §5.3 cannot be attributed to the claimed mechanism.","section":"§5.2–5.3"},{"comment":"The use of GPT-4 for both context retrieval and the fact-checking verification step introduces a self-referential evaluation. Because the same model provides the reference context and then judges whether a span is hallucinated, shared systematic biases could inflate the agreement rates (83% English, 58% Arabic). The manuscript should at least discuss this limitation and ideally verify a subsample with an independent retrieval-backed fact-checker.","section":"§4.1 and §5.3"}],"minor_comments":[{"comment":"The Arabic Cor is reported as 2.1 in the conclusion but 0.21 in §5.3; since a correlation coefficient cannot exceed 1 in magnitude, the conclusion should be corrected.","section":"§6"},{"comment":"There are several typos: 'Expriments' should be 'Experiments', 'mertrics' should be 'metrics', 'algirthm' should be 'algorithm', and 'languauge' should be 'language'.","section":"§5 heading and throughout"},{"comment":"The phrase 'hallucination detection at the spectral level' appears to be a typo for 'span-level'.","section":"§2"},{"comment":"The Arabic example strings appear garbled due to font/encoding issues; please ensure the Unicode text renders correctly in the final PDF.","section":"§3.1"},{"comment":"The text references Figure 1 and its workflow but does not walk through the figure's components; a short step-by-step explanation would improve readability.","section":"§4 and Figure 1"}],"recommendation":"major_revision","confidential_remarks":"This paper reads as a shared-task system description, and the most serious problem is the degenerate confidence score in Eq. (1). I would encourage the editor to verify that the authors correct this equation and rerun the experiments before considering publication. The lack of reproducibility (no α value, no threshold-selection procedure, no baselines) is also concerning and should be addressed in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a shared-task system description that has one genuinely useful result buried under a vacuous confidence mechanism. Read it for the Arabic parsing comparison; don't read it as evidence that token-level logits refine entailment scores.\n\nWhat's new: it applies SRL decomposition plus DeBERTa NLI to the Mu-SHROOM benchmark for English and Arabic. The only controlled comparison is Arabic SRL extraction: HanLP multilingual SRL gives IoU 0.205, while CamelParser dependency parsing followed by a custom SRL algorithm gives 0.28. That improvement is plausible and potentially useful to people working on Arabic hallucination detection. The LLM fact-checking (GPT-4 and LLaMA) is a reasonable auxiliary check, though using GPT-4 to retrieve context and verify spans makes agreement partly self-referential.\n\nThe soft spot is load-bearing and, unfortunately, mathematical. Equation (1) defines logit_score as the average of softmax probabilities computed over the tokens of the same atomic unit. Since the denominator is the sum over those same n tokens, the softmax values sum to 1, and the average is exactly 1/n. The paper's own example confirms it: a 3-token unit gets 0.333 and a 9-token unit gets 0.111. So the \"confidence\" term carries no information about the model's uncertainty, only about unit length. Equation (2) therefore reduces to alpha*entailment + (1-alpha)/n. The abstract's claim that entailment scores are refined by token-level confidence is unsupported. All reported IoU values (0.358 English, 0.28 Arabic) are just measurements of the entailment-plus-length combination. There are also no baselines, no error bars, no alpha value, and no code, so the numbers aren't independently checkable.\n\nMinor issues: the conclusion says Arabic Cor is 2.1 while the results section says 0.21; that's likely a typo. The references are a bit thin in places.\n\nProportionately, this is a workshop-style system paper, not a research claim. The Arabic parser comparison might survive as a small empirical note. But as it stands, the central mechanism doesn't work, and the paper needs at least a corrected confidence term and a baseline comparison before I'd take its headline claim seriously.\n\nI wouldn't send this to peer review in its current form; I'd desk reject and invite a resubmission if the authors fix the math. If you work on Mu-SHROOM or Arabic hallucination detection, the CamelParser result is worth a one-line citation, but nothing else here changes how I'd approach the problem.","headline":"A real flaw in the confidence equation empties the central claim, but the Arabic parser comparison is a decent empirical data point.","tokens_in":7408,"tokens_out":3314,"would_cite":false,"duration_ms":28711,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that hallucinated spans in LLM answers can be detected by decomposing each answer into semantic roles, scoring each role's entailment against retrieved context, and thresholding the combined score at 0.5.","keywords":["hallucination detection","span-level detection","semantic role labeling","textual entailment","token confidence","Arabic NLP","Mu-SHROOM","SemEval-2025"],"falsifier":"Recompute Eq. (1) for two units of the same length whose logits are opposite extremes: the confidence scores are identical, so the term cannot reflect token-level uncertainty; rerunning the pipeline with a per-token softmax average would change refined scores and the reported IoU if the confidence component were doing any work.","tokens_in":6481,"feed_emoji":"🔍","tokens_out":16345,"duration_ms":137230,"temperature":0.7,"pith_summary":"The paper claims that hallucinated spans in LLM-generated answers can be localized by decomposing the answer into atomic semantic roles, retrieving a reference context with GPT-4, and scoring each role with a DeBERTa textual entailment model. It blends the entailment probability with a token-level confidence score computed from the model's output logits, then labels any semantic unit whose refined score falls below 0.5 as hallucinated. On the Mu-SHROOM shared-task data, the framework reports an IoU of 0.358 and a correlation of 0.322 for English, and an IoU of 0.28 for Arabic when dependency parsing precedes semantic role labeling. The motivation is that sentence-level or token-level detectors miss which exact span is unsupported, while role-level units give a localizable target for correction and fact-checking.","feed_headline":"Semantic role check pinpoints hallucinated spans","feed_subtitle":"Role-level checks localize false spans in both languages, hitting 0.358 overlap in English and 0.28 in Arabic.","key_machinery":"The central object is the refined score per semantic unit, $\\text{refined\\_score} = \\alpha \\cdot \\text{entailment} + (1-\\alpha) \\cdot \\text{confidence}$, thresholded at 0.5 to mark hallucinated spans. The entailment component comes from a DeBERTa model classifying each SRL-extracted unit against the retrieved context; the confidence component is Eq. (1), defined as the average over the unit's $n$ tokens of $e^{\\logit_i}/\\sum_j e^{\\logit_j}$. The framework's other load-bearing machinery is the decomposition step: a BERT-based SRL model for English, and either a multilingual SRL model or CamelParser2.0 dependency parsing followed by SRL for Arabic, which create the atomic units that make span-level localization possible.","core_discovery":"On the paper's own account, hallucination is a property of semantic roles, not whole sentences: an LLM answer is decomposed into predicate-argument units, each unit is compared with a GPT-4-retrieved reference context using a DeBERTa entailment model, and the entailment probability is blended with a token-level logit confidence score to give a refined score per unit. Units whose refined score falls below 0.5 are labeled hallucinated spans. The paper reports that this pipeline achieves an IoU of 0.358 and a correlation of 0.322 on English Mu-SHROOM data, and that Arabic detection improves from IoU 0.205 with a multilingual SRL model to 0.28 when CamelParser dependency parsing is applied first. A separate LLM fact-checking step confirms 83% of the English hallucinated spans with GPT-4 and 72% with LLaMA.","pith_inferences":["One consequence the paper leaves implicit: Eq. (1) evaluates to one divided by the number of tokens in the unit, so the confidence term carries no per-token uncertainty and refined-score differences come from the entailment term and unit length alone.","A testable replacement, not proposed in the paper, is to use the mean per-token softmax probability as the confidence term and re-measure IoU on the same data; unchanged scores would show the logit component is inert, while changed scores would reveal that unit length was the real driver.","An extension the paper does not draw: the Arabic gain from adding dependency parsing before SRL (IoU 0.205 to 0.28) suggests that parser choice, not the entailment model, is the binding constraint for morphologically rich languages."],"forward_implications":["A unit whose refined score falls below 0.5 is treated as hallucinated, so the framework turns a continuous entailment/confidence signal into an explicit, inspectable span decision.","English detection reaches IoU 0.358 and correlation 0.322 on Mu-SHROOM, so predicted spans overlap the annotated hallucinated spans at a level that supports span-level use.","Arabic detection improves from IoU 0.205 with multilingual SRL alone to IoU 0.28 when CamelParser dependency parsing precedes SRL, implying that syntactic structure helps hallucination detection in morphologically rich languages.","Independent LLM fact-checking confirms 83% of English hallucinated spans with GPT-4 and 72% with LLaMA, indicating that the detected spans align with external factual judgment.","Because hallucinations are flagged per semantic unit, a downstream system could rewrite or correct only the offending span rather than the whole answer."],"supporting_citations":[{"why":"Provides the Mu-SHROOM dataset, task definition, and the IoU/Cor metrics the evaluation reports.","marker":"Vázquez et al., 2025"},{"why":"Supplies the DeBERTa textual entailment model that scores each semantic unit against the retrieved context.","marker":"He et al., 2021"},{"why":"Supplies the BERT-based SRL model used to decompose English answers into predicate-argument units.","marker":"Shi and Lin, 2019"},{"why":"Supplies CamelParser2.0, the dependency parser whose use before SRL raises Arabic IoU to 0.28.","marker":"Elshabrawy et al., 2023"},{"why":"Supplies the multilingual SRL extraction model used in the first Arabic experiment.","marker":"He and Choi, 2021"}],"fun_headline_variants":["Role-level entailment scores expose hallucinated spans","SRL + DeBERTa: finding false spans in LLM answers","CamelParser lifts Arabic hallucination detection to IoU 0.28","Span-level LLM hallucination: roles beat sentence-level checks","IoU 0.358: role-based detection of hallucinated spans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the logit-based confidence score measures how uncertain the model was about each token, even though the formula makes that score equal to one divided by the number of tokens in the unit.","fun_headline_variants_meta":{"raw":{"variants":["Role-level entailment scores expose hallucinated spans","SRL + DeBERTa: finding false spans in LLM answers","CamelParser lifts Arabic hallucination detection to IoU 0.28","Span-level LLM hallucination: roles beat sentence-level checks","IoU 0.358: role-based detection of hallucinated spans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1665,"prompt_tokens":883,"completion_tokens":782,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":706}},"tokens_in":499,"tokens_out":782,"duration_ms":7675,"temperature":1.0,"reasoning_tokens":706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:12:51.307065+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute Eq. (1) for two units of the same length whose logits are opposite extremes: the confidence scores are identical, so the term cannot reflect token-level uncertainty; rerunning the pipeline with a per-token softmax average would change refined scores and the reported IoU if the confidence component were doing any work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies CamelParser2.0, the dependency parser whose use before SRL raises Arabic IoU to 0.28."}],"review_version":1}