{"id":"34a6cb10-8456-4356-8ee8-7bfaf4c053b3","arxiv_id":"2608.10835","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"UniProbe detects hallucinated tokens in frozen LVLMs by jointly modeling attention graphs, image geometry, and response order, and uses this signal to resample bad tokens during generation.","lead":"UniProbe is a lightweight detector that reads a vision-language model's internal attention and hidden states to identify which words in its response are not supported by the image. It outperforms existing detectors at token-level hallucination detection and, during decoding, cuts object hallucinations by up to 55% while adding only 6% latency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on a single-mid-layer attention signature for hallucinated tokens (Sec. 4, Eq. 2) that is never measured directly; ablations show the graph matters, but a direct diagnostic is needed before the SOTA and 55% mitigation numbers can be taken at face value.","rationale":"This paper is a genuine attempt and has strong internal consistency: the ablations are informative, the gains are consistent across multiple backbones and benchmarks, and a small human preference study supports the mitigation story. I do not believe the work should be rejected. However, the single most load-bearing premise—the existence of a usable attention signature in one mid-layer trace—is not directly evidenced; it is inferred from ablation improvements. The reader's weakest assumption captures this, and the absence of released code/checkpoints plus validation-based selection of layer and threshold mean the reported margins could be optimistic. A direct attention-separability diagnostic would settle whether the mechanism is real, and would also clarify whether the streaming, prefix-only setting preserves the signal. The verdict should remain conditional pending that check and the standard reproducibility artifacts.","tokens_in":15687,"tokens_out":19349,"duration_ms":227072,"concrete_test":"On a held-out split of MHALO (and on the self-generated COCO captions used for Table 4), stratify response tokens by ground-truth hallucination label and compute per-token attention-mass features (sum and entropy over image, query, and response neighbors, plus top-k patch overlap) from the selected layer. Fit a simple logistic classifier on these attention-only features and report token-level AUC/F1. If attention-only features cannot separate classes above chance, the Eq. 2 signal is absent and the graph's gain must come from elsewhere; if they separate, the central premise is directly confirmed. As a control, repeat with the read-out layer and decoding threshold fixed on a development set before computing the reported test numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"UniProbe's GNN (Eq. 2) uses raw attention weights as edge strengths to propagate evidence from image patches, query tokens, and earlier response tokens, and the graph-construction paragraph prunes to the most-attended nodes. The relational advantage therefore depends on hallucinated response tokens having a distinguishable attention signature at the chosen mid-layer. This is asserted from prior work (Frasca et al., 2026) and supported only indirectly by Table 5b, where removing the graph costs 10.1 F1M; no measurement in the paper shows that the attention distributions of hallucinated and grounded tokens actually separate, nor that the separation survives the per-backbone layer choice and the streaming prefix-only setting. If the signature is weak or confounded by token position, length, or response style, the GNN degenerates toward the flat-probe baseline (31.9 F1M, Table 5a), and the claimed SOTA margins, already reported without error bars, are not established. The same premise is load-bearing for the 55% mitigation claim, because the streaming guardrail relies on the same scores; the paper's validation-selected read-out layer and threshold further raise the risk of optimistic reporting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UniProbe, a lightweight token-level hallucination detector for large vision-language models (LVLMs) that operates on a frozen backbone. From a single forward pass, it reads the hidden states and attention weights at a mid-network layer, builds a directed graph over image patches, query tokens, and response tokens, and processes this graph with alternating GNN, ViT, and GRU modules. A streaming variant with a unidirectional GRU enables hallucination-aware decoding that rejects and resamples flagged tokens. A self-adaptation procedure fine-tunes the detector on the target model's own CHAIR-labeled generations. Experiments on MHALO, HalLoc, POPE, and COCO captions across five backbones report state-of-the-art token-level and object-level hallucination detection, and a 55% reduction in object hallucinations at 1.06× vanilla latency in streaming decoding.","tokens_in":15977,"tokens_out":9053,"duration_ms":89897,"significance":"If the reported results hold, UniProbe is a significant contribution to trustworthy multimodal AI. It shows that a single lightweight probe over a frozen model's internal trace can localize hallucinated tokens with higher accuracy than full-model fine-tuning (HaloDet) and external verifiers (HalLocalizer), and that the same detector can drive decoding-time mitigation that reduces object hallucinations by more than half at a small latency cost. The paper is commendable for its breadth: five backbones, three tasks, extensive architecture ablations (Table 5 and Appendix D), and a human preference study. The method is reference-free, preserves the backbone, and the self-adaptation strategy directly addresses an important deployment shift. The systematic ablation isolating the contributions of the GNN, ViT, and GRU is a particular strength. However, the empirical claims rest entirely on point estimates with no error bars or significance tests, and the core mechanistic premise—that hallucinated tokens leave a distinguishable attention signature at the chosen mid-layer—is never directly verified. These gaps prevent the results from being fully convincing at this stage.","major_comments":[{"comment":"The method's central premise is that hallucinated response tokens have a distinguishable attention signature at the chosen mid-layer, particularly in how they attend to image patches, query tokens, and earlier response tokens. The paper never directly measures this separation. The ablation in Table 5b shows that removing the graph costs 10.1 F1M, but it does not establish that the attention weights themselves, rather than the GNN's inductive bias over hidden-state features, are the source of the gain. Please report a direct diagnostic, for example the distribution of total attention mass to image patches, the entropy of the per-modality attention masses c_i in Eq. (3), or the AUROC of a simple attention-based statistic for hallucinated versus grounded tokens at the read-out layer, across backbones and token positions. Without such evidence, the central mechanism is plausible but unverified, and the layer sweep in Fig. 4 could be selecting a layer for reasons unrelated to attention.","section":"Sec. 4, Eq. (2) and graph-construction paragraph"},{"comment":"All main results are presented as point estimates with no standard deviations, confidence intervals, or significance tests, despite the appendix mentioning 'seed noise' in Appendix D. Given that headline margins (for example, 4–6 F1M over HaloDet in Table 1 and 22.1 F1 over Token Grounding in Table 3) support the state-of-the-art claim, and that the read-out layer and decoding threshold are selected on validation data, the paper should report mean ± standard deviation over at least three seeds for the main comparisons and a paired significance test (for example, bootstrap or paired t-test) for the differences against the strongest baseline. This is especially important for the 55% mitigation claim in Table 4, where a single CHAIRi value could be within run-to-run variance.","section":"Tables 1–5 and Appendix D"},{"comment":"The evaluation does not specify whether the trained baselines (HaloDet, HalLocalizer, Whitehead et al., PAS, and the 'FT on self-generation' variants) were retrained by the authors under matched conditions or whether numbers were taken from prior publications. For example, Table 1 reports HaloDet at 55.4/59.1 F1M on Qwen-3-VL and GLM-4V backbones, but HaloDet was originally trained on a different backbone; the fine-tuning data, epochs, learning rate, and any adaptation to the new backbone are not described. Similarly, Table 3's POPE protocol is only referenced to Nguyen et al. (2026) and Jiang et al. (2025), and Table 4's 'FT on self-generation' is a single line without details. Please state for each baseline whether it was re-run, with which hyperparameters, and provide the training protocol or an exact reference to the implementation used.","section":"Section 5 (Evaluation) and Table 1"},{"comment":"The self-adaptation procedure labels hallucinated object mentions using CHAIR on 500 Objects365 images, and the subsequent evaluation in Table 4 uses CHAIR on COCO captions as the metric. Training on the same heuristic used for evaluation risks overfitting to CHAIR's specific false-positive and false-negative patterns (for example, synonym mismatches or unannotated objects). In addition, the paper does not explain how the object-level CHAIR labels are converted to token-level training targets for the token-level detector. Please specify the token-labeling procedure, and include an analysis of the agreement between CHAIR labels and human judgments, or at least a discussion of how robust the reported reductions are to CHAIR's labeling noise.","section":"Sec. 5, 'Mitigating Distribution Shift' and Table 4"},{"comment":"The read-out layer ℓ and decoding threshold τ are selected on validation data, which is standard, but the paper should report the sensitivity of the final headline metrics to these choices. Fig. 4 shows the layer curve for GLM-4.1V and LLaVA-1.5, but not the corresponding detection F1 at layers near the peak on the actual test sets; Fig. 5 shows CHAIRi versus τ but not the detection F1 or content-preservation trade-off around τ=0.70. The reported 55% reduction may depend strongly on the validation-based τ choice; please provide a small sensitivity table (for example, F1 obj, CHAIRi, and correct-object coverage for τ = 0.60, 0.70, and 0.80).","section":"Sec. 6 and Fig. 5"}],"minor_comments":[{"comment":"The text mentions a 'Project Page' but provides no URL or linked identifier; please add the complete URL or a footnote.","section":"Abstract and Section 1"},{"comment":"Please use 'LLaVA-1.5' consistently instead of 'LLaV A-1.5' in the table headings and text.","section":"Tables 2 and 3"},{"comment":"The definitions of F1M and F1IoU are brief; please clarify that F1M is a word-overlap F1 at the token level and specify the exact matching rules (for example, case-insensitive matching, stemming, and how partial overlaps are handled).","section":"Section 5 (Datasets and metrics)"},{"comment":"The ViT module is described as operating 'over the image grid'; please clarify that it operates on the selected image patch nodes arranged by their 2-D coordinates, rather than on the full image grid.","section":"Section 4, Eq. (4)"},{"comment":"The claim that 'the backbone verbally denies an object that its own internal state encodes as present' is interesting but not quantitatively supported; consider adding a small analysis of such cases.","section":"Appendix C"},{"comment":"The user study reports preference percentages without confidence intervals or inter-rater agreement statistics; please add 95% confidence intervals and a measure such as Cohen's kappa.","section":"Appendix B"},{"comment":"The paper states that all hyperparameters were selected through ablations on held-out validation data, but several important budgets (N_img, N_qry, k_img, k_qry, k_resp) are only ablated in Appendix D; consider summarizing these results in the main text.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"This is a technically strong paper with a novel architecture and a substantial evaluation across backbones and tasks. The main concern is that the reported gains, including the headline 55% mitigation figure, are not accompanied by error bars or significance tests, which is important because the margin over baselines in some comparisons is claimed despite seed noise elsewhere. The mechanistic premise about attention-based separation should be made explicit with a direct measurement, as it is load-bearing for the method's justification. I believe the paper is within the scope of the journal and the issues are addressable. I recommend major revision rather than rejection, and I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"UniProbe is a solid, well-ablated extension of the structured-trace line (CHARM, ACT-ViT, LOS-Net) from text-only LLMs to LVLMs. The new piece is a heterogeneous graph over image patches, query tokens, and response tokens, with attention weights as edges, processed by alternating GNN, ViT, and GRU blocks. That is a real architectural novelty, and the paper does it carefully: multiple backbones, three tasks, ablations of each module, a read-out layer sweep, a threshold trade-off, and a human preference study. The streaming variant with self-adaptation on Objects365+CHAIR is a practical answer to the distribution-shift problem, and the 55% CHAIR_i reduction at 1.06x latency is the headline result. The paper also honestly lists its limitations: open-source only, per-backbone readouts, no cross-backbone transfer.\n\nThe soft spots are reproducibility and statistical rigor. Every number is a point estimate with no standard deviation or significance test. The read-out layer and decoding threshold are selected on validation data, so there is a real risk of optimistic selection bias. No code or checkpoints are released. Baseline training details, especially for POPE and the streaming comparisons with other detectors, are incomplete. These are fixable in revision.\n\nOne deeper concern: the graph mechanism rests on the premise that hallucinated tokens have a distinguishable attention signature at the chosen mid-layer. The paper never measures that directly. The ablation removing the graph costs 10 F1M, which is strong indirect evidence, but a direct diagnostic—comparing attention distributions of hallucinated versus grounded tokens at that layer, and showing they separate—would make the mechanism much more convincing. As it stands, the graph could be helping in a different way. This is not a deal-breaker, but it is a missing measurement in a paper whose central claim is that structure matters.\n\nWho is this for? Anyone working on LVLM trustworthiness, hallucination detection, or decoding-time mitigation. It deserves a serious referee: the empirical scope is broad, the ablations are honest, and the architecture is likely to be reused. My recommendation: send to peer review, and ask for error bars, code release, and ideally a direct analysis of the attention separation. Without those, I would not yet treat the SOTA numbers as a benchmark.","headline":"A credible, well-ablated extension to LVLMs; the SOTA numbers are plausible but need error bars and a direct test of the attention premise.","tokens_in":16532,"tokens_out":4110,"would_cite":true,"duration_ms":39095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight probe that reads one mid-network layer's attention and hidden states can localize hallucinated tokens in frozen LVLMs and, when streamed during decoding, resample them away, cutting object hallucinations by up to 55% at…","keywords":["token-level hallucination detection","large vision-language models","attention graph","internal representations","hallucination-aware decoding","self-adaptation","computational trace","frozen backbone"],"falsifier":"Take a backbone and construct a test set where hallucinated tokens are deliberately generated to attend to image patches exactly as strongly as grounded tokens, for instance by conditioning the model on an image that contains the mentioned object but in a different attribute, so the object is present yet the mention is still wrong. If UniProbe's token-level F1 on this set drops to near the flat-probe level, the attention-signature assumption is falsified; if it stays high, the detector is reading a different cue.","tokens_in":15514,"feed_emoji":"🛡️","tokens_out":6338,"duration_ms":58602,"temperature":0.7,"pith_summary":"UniProbe claims that token-level hallucination detection in large vision-language models is best treated as a structured-data problem: one forward pass of a frozen model yields hidden states and attention weights, which UniProbe arranges into a directed graph over image patches, query tokens, and generated response tokens. Alternating GNN, ViT, and GRU blocks let relational, spatial, and sequential evidence interact, and a linear head scores each response token. On MHALO and HalLoc the detector surpasses trained baselines that fine-tune the backbone, and a streaming version used during decoding rejects and resamples flagged tokens, cutting object hallucinations by up to 55% at 1.06x vanilla latency. The paper further shows that adapting the detector to the host model's own free-form generations closes a train-test distribution shift that otherwise degrades precision.","feed_headline":"Frozen-model probe cuts object hallucinations by 55%","feed_subtitle":"UniProbe reads one layer's attention and hidden states, flags unsupported tokens, and resamples them at 1.06x latency.","key_machinery":"The computational-trace graph is the load-bearing object: each response token keeps its top attention edges to image patches, query tokens, and earlier response tokens, with the attention weight as edge weight and a per-modality attention-mass summary c_i (Eq. 3), so the detector reads 'attends to image' and 'attends to query' as distinct evidence. L alternating blocks apply a GNN (Eq. 2) to mix evidence across modalities, a ViT (Eq. 4) over the 2-D image grid, and a BiGRU (Eq. 5) along response order; a linear head (Eq. 6) gives per-token probabilities. The streaming variant swaps the BiGRU for a unidirectional GRU so scores depend only on the prefix, and reject-and-resample decoding bans flagged tokens. The paper identifies a mid-network 'grounding band' (layer 20 of GLM-4.1V, layer 14 of LLaVA-1.5) where detection peaks, and prunes the graph to at most 200 image and 64 query nodes, keeping the probe at about 16M parameters.","core_discovery":"The central claim is that a lightweight probe over a single mid-network layer's attention and hidden states can localize hallucinated tokens more accurately than full-model fine-tuning or external verifiers, and that the same probe, made causal, can act as a decoding guardrail. Concretely, UniProbe builds a typed directed graph in which response tokens draw edges to the most-attended image patches, query tokens, and earlier response tokens, with attention weights as edge features; three interleaved structure-aware modules update a shared per-token representation, and a linear head outputs per-token hallucination probabilities. The paper reports token-level F1 of 63.2/52.9 (F1M/F1IoU) on MHALO with GLM-4V, surpassing HaloDet's 59.1/49.8; object-hallucination F1 of 63.1 on POPE versus 41.0 for the strongest prior; and, with self-adaptation, a 55% reduction in CHAIRi at 1.06x latency while preserving caption quality and output distribution.","pith_inferences":["If the mid-layer grounding band is a general property of LVLMs, the read-out layer could be chosen automatically by a lightweight probing sweep, and the same graph construction might transfer to other fine-grained faithfulness tasks such as evidence attribution or chain-of-thought verification.","The ablation result that flat MLP and Transformer probes trail the graph-based probe by at least 30 F1M suggests that the signal is relational: hallucinated tokens are not identifiable from isolated hidden states but from how they route attention to context.","A testable extension: applying the same reject-and-resample guardrail to attribute and relationship hallucinations, not just object mentions, may require richer labels than CHAIR, since CHAIR only captures object presence.","The paper's threshold analysis implies an operating curve: tightening tau below 0.70 removes more hallucinations but starts deleting correct content, so deployment needs a content-preservation constraint, not just a hallucination target."],"forward_implications":["Backbone-freezing detection: a single forward pass of the frozen LVLM provides enough signal to beat fine-tuned taggers, so safeguards no longer require retraining or degrading the protected model.","Decoding-time prevention: the same detector, streamed causally, turns detection into prevention by rejecting and resampling hallucinated tokens before they enter the final response.","Distribution-shift correction: self-adaptation on the host model's own free-form captions, labeled automatically with CHAIR, lifts streaming detection F1 from 32.6 to 63.8 and cuts CHAIRi by 55%.","Latency budget: scoring a full response costs 1.15x vanilla latency and the streaming guardrail 1.06x, below all trained baselines, so the accuracy gain does not come at prohibitive compute.","Per-backbone readouts: internal representations differ across models, so the paper trains a separate readout per backbone and peaks the read-out layer by validation F1."],"supporting_citations":[{"why":"Supplies the attention-graph message-passing method and the pruning rule for keeping high-attention context.","marker":"(Frasca et al., 2026)"},{"why":"Shows activation tensors can be read by a ViT, motivating the structure-aware read-out.","marker":"(Bar-Shalom et al., 2025)"},{"why":"Provides the MHALO benchmark, the HaloDet fine-tuning baseline, and the Analyze-then-Judge zero-shot protocol.","marker":"(Cai et al., 2025)"},{"why":"Provides the HalLoc benchmark and the HalLocalizer external-verifier baseline it must beat.","marker":"(Park et al., 2025)"},{"why":"CHAIR metric used to auto-label self-generated captions for self-adaptation.","marker":"(Rohrbach et al., 2018)"},{"why":"Token Grounding, the strongest prior object-detection baseline on POPE and COCO.","marker":"(Nguyen et al., 2026)"},{"why":"Lookback Lens, the attention-based detector-guided decoding scheme that the reject-and-resample guardrail follows.","marker":"(Chuang et al., 2024)"},{"why":"HALP, a global internal probe baseline that UniProbe's token-level read-out extends.","marker":"(Kogilathota et al., 2026)"}],"fun_headline_variants":["UniProbe cuts object hallucinations by 55% at 1.06x latency","Lightweight probe slashes hallucinations 55% without full-model tuning","Token-level detector resamples hallucinations during generation, cutting them 55%","55% fewer object hallucinations with a frozen-model probe","Single-pass probe learns to spot and fix hallucinations in LVLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Hallucinated tokens leave a distinguishable trace in the attention a single mid-network layer pays to image patches, query tokens, and earlier response tokens; if a hallucinated token attends to the image as strongly as a grounded one, the graph message passing loses its primary signal and the detector's reported gains would not survive.","fun_headline_variants_meta":{"raw":{"variants":["UniProbe cuts object hallucinations by 55% at 1.06x latency","Lightweight probe slashes hallucinations 55% without full-model tuning","Token-level detector resamples hallucinations during generation, cutting them 55%","55% fewer object hallucinations with a frozen-model probe","Single-pass probe learns to spot and fix hallucinations in LVLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1691,"prompt_tokens":1054,"completion_tokens":637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":543}},"tokens_in":670,"tokens_out":637,"duration_ms":7242,"temperature":1.0,"reasoning_tokens":543,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:23:14.073969+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a backbone and construct a test set where hallucinated tokens are deliberately generated to attend to image patches exactly as strongly as grounded tokens, for instance by conditioning the model on an image that contains the mentioned object but in a different attribute, so the object is present yet the mention is still wrong. If UniProbe's token-level F1 on this set drops to near the flat-probe level, the attention-signature assumption is falsified; if it stays high, the detector is reading a different cue.","supporting_citations":[{"cited_title":"MHALO : Evaluating MLLM s as Fine-grained Hallucination Detectors","cited_arxiv_id":null,"evidence_quote":"Provides the MHALO benchmark, the HaloDet fine-tuning baseline, and the Analyze-then-Judge zero-shot protocol."},{"cited_title":"Object Hallucination in Image Captioning","cited_arxiv_id":null,"evidence_quote":"CHAIR metric used to auto-label self-generated captions for self-adaptation."},{"cited_title":"HALP : Detecting Hallucinations in Vision-Language Models without Generating a Single Token","cited_arxiv_id":null,"evidence_quote":"HALP, a global internal probe baseline that UniProbe's token-level read-out extends."}],"review_version":1}