Pith. sign in

REVIEW 5 major objections 6 minor 6 cited by

The paper claims that the two main sources of noise in token-level visual attributions for multimodal LLMs—fragmented patches and contamination from preceding text tokens—can be corrected post-hoc by multi-scale evidence fusion and rank-bas

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 14:52 UTC pith:BQQHAQ2Y

load-bearing objection A plausible incremental refinement for token-level attribution, with consistent gains but tuning-selection and mechanism-specification issues that likely inflate the headline numbers. the 5 major comments →

arxiv 2509.22415 v4 pith:BQQHAQ2Y submitted 2025-09-26 cs.CV cs.AI

Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models

Jiawei Liang , Jianjie Huang , Ruoyu Chen , Xianghao Jiao , Siyuan Liang , Shiming Liu , Xiaochun Cao This is my paper
classification cs.CV cs.AI
keywords visual attributionmultimodal large language modelslogit lensmulti-scale aggregationtoken interactioninterpretabilityRank-Biased Overlapcontext residualization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that visual attribution maps for multimodal large language models are noisy for two specific, fixable reasons: each visual token is decoded in isolation, losing spatial context, and the autoregressive context inherits spurious activations from preceding tokens. It introduces ERCR, which aggregates logit-lens maps over several input resolutions (MSEA) and subtracts a fitted combination of preceding-token maps weighted by rank-biased overlap (ARC). On Qwen2-VL-2B it reports F1-IoU rising from 39.10 to 44.45 on COCO Caption and from 30.83 to 37.20 on GranDf, with consistent gains across LLaVA, Qwen2-VL, and InternVL variants of 2B–13B. A sympathetic reader would care because the method is training-free and post-hoc, so it can be dropped into existing models without retraining.

Core claim

The central claim is that existing logit-lens attribution—decoding each visual token's hidden state through the unembedding layer—fails in two systematic ways: token-level decoding ignores spatial context, and the target token's activation is contaminated by preceding text tokens. ERCR addresses the first by re-reading the image at several scales and averaging the resulting attribution maps, giving each visual location a dynamic receptive field; it addresses the second by computing a context map from the preceding tokens' own attribution maps, weighting each such map by (1−r_j) where r_j is the Rank-Biased Overlap between the top-k token predictions of that context token and the target, then

What carries the argument

Two complementary mechanisms: Multi-Scale Explanation Aggregation (MSEA), which resizes the input image, re-tokenizes it, reads out logit-lens scores at each scale, and fuses the rescaled maps by averaging; and Activation Ranking Correlation (ARC), which defines a context token's interference weight as one minus the Rank-Biased Overlap between its top-50 predicted token rankings and the target token's rankings, forms a context map by weighting each preceding token's attribution map accordingly, and subtracts a fitted multiple (β) of that map from the target map before positive clipping and a rank Gaussian filter. MSEA supplies spatial context; ARC suppresses context-token interference.

Load-bearing premise

The load-bearing premise is that interference from preceding text tokens contaminates the current token's attribution additively, as a weighted sum of those tokens' own attribution maps with weights given by rank-biased overlap, and that subtracting a single fitted multiple of this sum removes noise without removing real evidence.

What would settle it

Take a fixed image and a generated caption, compute ERCR's residualized map, then recompute it with the caption's context tokens shuffled or deleted before the target token. If the map barely changes, the residualization is not needed. Also compare a version of ARC with permuted (random) RBO weights: if F1-IoU does not drop back to the un-residualized baseline, then the ranking signal is not the operative part of the method.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • F1-IoU improves by 5.35 points on COCO Caption and 6.37 points on GranDf for Qwen2-VL-2B relative to the prior logit-lens method, driven mainly by a 23-point reduction in functional false positives.
  • The method is post-hoc and training-free, so it can be applied to any off-the-shelf MLLM without altering weights or requiring ground-truth masks at inference time.
  • Across seven model variants, gains are consistent and tend to grow with model scale (from 11.53 to 14.52 percentage points in InternVL's F1-IoU), suggesting the technique leverages robust statistical properties of learned representations.
  • MSEA and ARC are complementary: the multi-scale fusion raises object-level IoU while the ranking-based residualization suppresses noise on non-semantic tokens, so using both gives more than either alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The core hypothesis—that preceding-token interference enters additively and can be estimated from the preceding tokens' own attribution maps—could be tested directly by deleting or permuting context tokens and checking whether the residualized map changes exactly as ARC predicts; this would separate the mechanism from the aggregation gain.
  • Because ARC only uses output logits and top-k rankings, the same residualization idea could transfer to other autoregressive decoders (e.g., text-only or speech models), where the 'visual tokens' would be replaced by any parallel input stream.
  • The paper defines a base attribution A0 but never uses it in the final formula, so the residualization as written is under-specified; reproducing the exact algorithm may require additional assumptions about which tokens are treated as the irrelevant set.
  • A synthetic test with one controlled known noise source (e.g., injecting a fixed activation pattern from a distractor token) would tell whether the RBO weights are identifying the true interferer or merely fitting noise; if random weights work as well, the ranking information is not the active ingredient.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes two post-hoc modifications to logit-lens visual attribution in multimodal LLMs. MSEA computes attribution maps at several input scales and fuses them, aiming to reduce fragmentation caused by the limited receptive field of a single visual token. ARC models interference from preceding text tokens by computing RBO rank similarity between top-k logit rankings, building a weighted map of preceding-token attributions, and subtracting a fitted component of this map from the current token's map. The method is evaluated on Qwen2-VL, LLaVA-1.5, and InternVL2.5 over COCO Caption, GranDf, and OpenPSG, using Obj-IoU, Func-IoU, and their harmonic mean F1-IoU. The main numerical claims are consistent improvements over TAM, e.g., F1-IoU rising from 39.10 to 44.45 on COCO Caption for Qwen2-VL-2B.

Significance. If the claims hold, the paper would offer a practical, training-free refinement for token-level visual attribution in MLLMs, with convincing breadth across architectures and scales. The strengths are the clear motivation, the wide empirical coverage (seven models, three datasets), and the fact that the method is purely post-hoc. However, the central improvement is carried by the ARC residualization step, whose underlying model is not independently validated, and by a metric (F1-IoU) that is tightly aligned with what ARC is designed to suppress. The absence of error bars and the selection of hyperparameters on the evaluation sets weaken the empirical backing. These issues are fixable but currently leave the main claim less secure than the tables suggest.

major comments (5)
  1. [§3.3, Eq. (8)–(9)] The ARC mechanism rests on an unvalidated additive-noise assumption: preceding-token interference is modeled as a weighted sum of individual attribution maps, with weights (1−r_j), and the component of A_t along \hat{A}_t is then removed by the least-squares fit β. If \hat{A}_t overlaps the true visual evidence for T_t — which is likely when preceding tokens refer to the same object or scene — Eq. (9) subtracts real evidence. The paper provides no diagnostic showing that \hat{A}_t is uncorrelated with the target ground-truth mask. I ask for a controlled experiment that distinguishes genuine noise suppression from a generic reduction of diffuse activations: e.g., compare against subtracting a random or constant map matched in total activation, and report Obj-IoU on target tokens as well as Func-IoU.
  2. [§4.1, Eq. (10)] Func-IoU is defined only verbally as 'activations on non-semantic tokens (e.g., punctuation), where higher values indicate fewer false positives.' No formula is given, and no specification of which tokens are treated as non-semantic is provided. Since Func-IoU contributes to the primary metric F1-IoU, and since ARC is explicitly designed to suppress diffuse activations on non-semantic tokens, the reported gains in Func-IoU may be partly by construction. The definition must be made precise, and the authors should report whether the improvement persists when non-semantic tokens are excluded or when a metric less aligned with the method's objective is used.
  3. [§4.2, Fig. 3 and Table 1] The hyperparameters appear to be selected on the same evaluation sets used for the headline results. Figure 3 reports that four scales, e.g., [0.5,0.75,1.0,1.25], yield peak performance, yet the main configuration in §4.1 and Table 1 uses three scales [0.5,0.75,1.0]. No error bars, variance estimates, or significance tests are reported. The authors should either justify the choice of three scales, or adopt the empirically better four-scale setting; in either case, reporting multiple random/order replicates with confidence intervals would make the improvements credible.
  4. [§3.3, Eq. (8)] The ARC algorithm is incompletely specified. A base attribution A_0 is defined just before Eq. (9) as the raw attribution for the vocabulary token with minimal visual activation, but it never appears in the refinement formula. The RBO metric in Eq. (7) requires a persistence parameter, which is not stated. It is also unclear whether the maps A_j are normalized before the weighted sum, and whether β in Eq. (9) is estimated per token or globally. These details are necessary to reproduce the method and to understand what exactly is subtracted.
  5. [Table 3] The ablation table is unreadable as printed. The header row mixes method labels ('Mean', 'Max', 'ARC', 'TAM') with the metric names, and the body rows do not clearly identify which configuration each row corresponds to. The parenthetical deltas such as '+10.21' and '+12.88' are not tied to a baseline. Since the claim that MSEA and ARC are complementary rests entirely on this table, it must be reformatted with explicit row and column labels.
minor comments (6)
  1. [Title/Abstract vs. §3] The paper title and submitted abstract describe 'Evidence Recomposition' and 'Predictive Context Residualization' (ERCR/PCR), but the method sections and experiments present MSEA and ARC. This inconsistency suggests a version-control problem and must be fixed.
  2. [Figure 1 caption] The caption says 'MSEA (a) and SAC (b),' but the method is called ARC. Please correct the acronym.
  3. [§3.2, Eq. (5)] The resize/aggregation step in Eq. (5) uses 1/α_s, but for models with fixed input size the padded image in Eq. (3) has already been embedded in a canvas of size H×W. It should be clarified how the padded regions are handled during resizing so that spatial coordinates remain consistent.
  4. [§3.3, Eq. (9)] The notation G(⌊A_t − β\hat{A}_t⌋_+) is confusing: the floor symbol is nonstandard for a ReLU-like operation, and 'Rank Gaussian Filter' G is never formally defined. Please define the post-processing and the positive-part operation explicitly.
  5. [Table 1] The baseline 'Archi.-Surgery 2025b' has no corresponding full reference in the bibliography, and the year/venue is unclear. Please add the missing reference.
  6. [Throughout] There are numerous spacing and typographical issues in the tables (e.g., 'Obj-IoUFunc-IoUF1-IoU') and in the text ('LLaV A-1.5' vs. 'LLaVA'). A careful proofreading pass is needed.

Circularity Check

0 steps flagged

No significant circularity: MSEA and ARC are empirical post-hoc transformations evaluated against external ground-truth benchmarks, and no load-bearing self-citation or fitted-parameter-as-prediction step is present.

full rationale

The paper's derivation is self-contained. MSEA (Eq. 5) simply averages logit-lens attribution maps across rescaled inputs; ARC (Eqs. 7-9) estimates a weighted combination of preceding-token attribution maps using RBO and subtracts a least-squares fitted multiple from the target map. The evaluation uses externally defined Obj-IoU, Func-IoU, and F1-IoU against ground-truth masks (Eq. 10), not quantities constructed from the method's own outputs. The per-token β is a projection coefficient fitted to minimize reconstruction error of A_t, not fitted to the benchmark labels, so it is not a fitted parameter renamed as a prediction. The paper relies on independent prior work (logit lens, TAM, RBO) rather than a self-citation chain. Two specification gaps exist—the defined base attribution A₀ is never used in Eq. 9, and the RBO persistence parameter is not stated—but these affect reproducibility, not circularity. Hyperparameter selection using the same benchmarks is a correctness/overfitting concern, not a circularity concern under the stated rules. Therefore the central claim does not reduce to its inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on the logit-lens paradigm, an additive-noise model of preceding-token interference, and several hyperparameters tuned on the evaluation benchmarks. No new physical or conceptual entities are introduced.

free parameters (4)
  • scale factors alpha_s = [0.5, 0.75, 1.0] main; sensitivity shows [0.5,0.75,1.0,1.25] peaks
    Selected based on sensitivity analysis on the same test sets (Fig. 3).
  • top-k k = 50
    Hand-set; no sensitivity analysis or justification.
  • residualization coefficient beta = per token, argmin ||A_t - beta*hat A_t||^2
    Fitted per token to minimize residual between target and context maps.
  • Rank Gaussian Filter G = unspecified
    Used for post-processing; parameters not given.
axioms (4)
  • domain assumption Logit-lens projection of visual hidden states into vocabulary space yields meaningful token-level attribution.
    From prior work (nostalgebraist; TAM), used in Eq. 2 without independent validation here.
  • ad hoc to paper Interference from preceding text tokens is linearly additive and representable as a weighted sum of their attribution maps, with weights from RBO rank alignment.
    Central to ARC residualization (Eq. 8-9); no independent evidence provided for this linear-noise model.
  • domain assumption RBO top-k ranking alignment measures semantic relevance of preceding tokens to the current token.
    Adopted from Webber et al. and applied in Eq. 7 without validation in this setting.
  • ad hoc to paper There exists a base attribution A0 from the token with minimal visual activation that can serve as a reference.
    Defined in Section 3.3 but never used in the final equation; role is unclear.

pith-pipeline@v1.3.0-alltime-deepseek · 12094 in / 12353 out tokens · 82730 ms · 2026-08-04T14:52:15.454535+00:00 · methodology

0 comments
read the original abstract

Multimodal large language models (MLLMs) have achieved strong vision-language performance, yet their token-level visual evidence remains difficult to inspect. Recent logit-lens attribution methods project each visual-token hidden state into the vocabulary space to explain generated words, but this token-wise readout introduces a mismatch: visual tokens are context-mixed by the model, while the attribution score is decoded independently at each token location. This often produces fragmented attribution maps and can be further affected by autoregressive context signals from preceding text tokens. We propose ERCR, an attribution framework built from Evidence Recomposition (ER) and Predictive Context Residualization (PCR). ER aggregates target evidence across multiple views with different token-to-region assignments, reducing attribution fragmentation caused by a single readout grid. PCR estimates a preceding-token context map with RBO-based rank relevance and subtracts its fitted component from the ER map to suppress context-token interference. Experiments on LLaVA, Qwen2-VL, and InternVL families across COCO Caption, GranDf, and OpenPSG show that ERCR improves visual evidence for target tokens and mitigates preceding-token context interference under the existing evaluation protocol. On Qwen2-VL-2B, ERCR improves TAM F1-IoU from 39.10 to 44.45 on COCO Caption and from 30.83 to 37.20 on GranDf. Overall, ERCR provides a practical refinement for token-level visual evidence inspection.

Figures

Figures reproduced from arXiv: 2509.22415 by Jianjie Huang, Jiawei Liang, Ruoyu Chen, Shiming Liu, Siyuan Liang, Xianghao Jiao, Xiaochun Cao.

Figure 1
Figure 1. Figure 1: Motivation of our proposed MSEA (a) and SAC (b). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our proposed framework. text token Tt is denoted by h l t . At the final layer, the hidden state of the current text token h L t is projected onto the vocabulary space through the unembedding matrix WU , yielding the logits. The next-token probability distribution is then given by P(Tt | T<t, I) = softmax(WU h L t ), (1) where h L t denotes the hidden state of the t-th text tokens at the L th l… view at source ↗
Figure 3
Figure 3. Figure 3: Performance sensitivity to the number and range of scaling factors across datasets and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of attribution maps generated using the Qwen2-VL-2B model. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of attribution maps generated using the LLaVA-1.5-7B model. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Correct Decisions Hide Internal Stress: Decision-State Probing in Multimodal Language Models

    cs.CL 2026-06 unverdicted novelty 7.0

    S³E framework finds excess decision-state displacement under semantic stress in multimodal models despite consistent correct forced-choice behavior.

  2. Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads

    cs.CL 2026-06 unverdicted novelty 7.0

    Multimodal LLMs exhibit functional sparsity where a small number of CoRe attention heads handle cross-modal retrieval, with ablation of the top 5% degrading performance while others have little effect.

  3. What if? Emulative Simulation with World Models for Situated Reasoning

    cs.CV 2026-03 conditional novelty 6.5

    WanderDream supplies 15.8K panoramic mental-exploration videos and 158K QA pairs showing that world-model imagination measurably improves situated spatial reasoning without active exploration.

  4. MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

    cs.LG 2026-04 unverdicted novelty 6.0

    MACS improves MoE MLLM inference efficiency via entropy-weighted token loads and dynamic modality-adaptive expert capacity allocation.

  5. MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

    cs.LG 2026-04 unverdicted novelty 6.0

    MACS improves inference speed in multimodal MoE models by entropy-weighted balancing of visual tokens and real-time modality-adaptive expert capacity allocation.

  6. MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

    cs.LG 2026-04 unverdicted novelty 5.0

    MACS reduces Expert Parallelism stragglers in MoE MLLMs via entropy-weighted visual token load and dynamic modality-adaptive expert capacity, without retraining.

Reference graph

Works this paper leans on

41 extracted references · 12 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Quantifying attention flow in transformers

    Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 4190--4197, 2020

    2020

  2. [2]

    Attnlrp: Attention-aware layer-wise relevance propagation for transformers

    Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Attnlrp: Attention-aware layer-wise relevance propagation for transformers. In International Conference on Machine Learning, pp.\ 135--168. PMLR, 2024

    2024

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 0 23716--23736, 2022

    2022

  4. [4]

    Xai for transformers: Better explanations through conservative propagation

    Ameen Ali, Thomas Schnake, Oliver Eberle, Gr \'e goire Montavon, Klaus-Robert M \"u ller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. In International Conference on Machine Learning, pp.\ 435--451. PMLR, 2022

    2022

  5. [5]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp.\ 2425--2433, 2015

    2015

  6. [6]

    Lvlm-intrepret: An interpretability tool for large vision-language models

    Gabriela Ben Melech Stan, Estelle Aflalo, Raanan Yehezkel Rohekar, Anahita Bhiwandiwalla, Shao-Yen Tseng, Matthew Lyle Olson, Yaniv Gurwicz, Chenfei Wu, Nan Duan, and Vasudev Lal. Lvlm-intrepret: An interpretability tool for large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8182--8187, 2024

    2024

  7. [7]

    Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks

    Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pp.\ 839--847. IEEE, 2018

    2018

  8. [8]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    Pith/arXiv arXiv 2015

  9. [9]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    A comprehensive survey of deep learning for image captioning

    MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR), 51 0 (6): 0 1--36, 2019

    2019

  11. [11]

    Interpreting and editing vision-language representations to mitigate hallucinations

    Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. arXiv preprint arXiv:2410.02762, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Layercam: Exploring hierarchical class activation maps for localization

    Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing, 30: 0 5875--5888, 2021

    2021

  13. [13]

    a ldchen, Alexander Binder, Gr \'e goire Montavon, Wojciech Samek, and Klaus-Robert M \

    Sebastian Lapuschkin, Stephan W \"a ldchen, Alexander Binder, Gr \'e goire Montavon, Wojciech Samek, and Klaus-Robert M \"u ller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10 0 (1): 0 1096, 2019

    2019

  14. [14]

    Token activation map to visually explain multimodal llms

    Yi Li, Hualiang Wang, Xinpeng Ding, Haonan Wang, and Xiaomeng Li. Token activation map to visually explain multimodal llms. arXiv preprint arXiv:2506.23270, 2025 a

    Pith/arXiv arXiv 2025

  15. [15]

    A closer look at the explainability of contrastive language-image pre-training

    Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, and Xiaomeng Li. A closer look at the explainability of contrastive language-image pre-training. Pattern Recognition, pp.\ 111409, 2025 b

    2025

  16. [16]

    Imitated detectors: Stealing knowledge of black-box object detectors

    Siyuan Liang, Aishan Liu, Jiawei Liang, Longkang Li, Yang Bai, and Xiaochun Cao. Imitated detectors: Stealing knowledge of black-box object detectors. In Proceedings of the 30th ACM International Conference on Multimedia, 2022

    2022

  17. [17]

    Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning

    Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. arXiv preprint arXiv:2311.12075, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Revisiting backdoor attacks against large vision-language models from domain shift

    Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Mingli Zhu, Xiaochun Cao, and Dacheng Tao. Revisiting backdoor attacks against large vision-language models from domain shift. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 9477--9486, 2025 a

    2025

  19. [19]

    T2vshield: Model-agnostic jailbreak defense for text-to-video models

    Siyuan Liang, Jiayang Liu, Jiecheng Zhai, Tianmeng Fang, Rongcheng Tu, Aishan Liu, Xiaochun Cao, and Dacheng Tao. T2vshield: Model-agnostic jailbreak defense for text-to-video models. arXiv preprint arXiv:2504.15512, 2025 b

    Pith/arXiv arXiv 2025

  20. [20]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp.\ 740--755. Springer, 2014

    2014

  21. [21]

    Agentsafe: Benchmarking the safety of embodied agents on hazardous instructions

    Aishan Liu, Zonghao Ying, Le Wang, Junjie Mu, Jinyang Guo, Jiakai Wang, Yuqing Ma, Siyuan Liang, Mingchuan Zhang, Xianglong Liu, et al. Agentsafe: Benchmarking the safety of embodied agents on hazardous instructions. arXiv preprint arXiv:2506.14697, 2025

    arXiv 2025

  22. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  23. [23]

    Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023

    MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023

    2023

  24. [24]

    Interpreting GPT : The logit lens

    nostalgebraist. Interpreting GPT : The logit lens. LessWrong, Aug 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  25. [25]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13009--13018, 2024

    2024

  26. [26]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp.\ 618--626, 2017

    2017

  27. [27]

    Releasing 3b and 7b redpajama-incite family of models including base, instruction-tuned & chat models

    Together.xyz. Releasing 3b and 7b redpajama-incite family of models including base, instruction-tuned & chat models. https://www.together.xyz/blog/redpajama-models-v1, 2023

    2023

  28. [28]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    A similarity measure for indefinite rankings

    William Webber, Alistair Moffat, and Justin Zobel. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS), 28 0 (4): 0 1--38, 2010

    2010

  30. [30]

    Genderbias- emph \ VL \ : Benchmarking gender bias in vision language models via counterfactual probing

    Yisong Xiao, Aishan Liu, QianJia Cheng, Zhenfei Yin, Siyuan Liang, Jiapeng Li, Jing Shao, Xianglong Liu, and Dacheng Tao. Genderbias- emph \ VL \ : Benchmarking gender bias in vision language models via counterfactual probing. arXiv preprint arXiv:2407.00600, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Fairness mediator: Neutralize stereotype associations to mitigate bias in large language models

    Yisong Xiao, Aishan Liu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. Fairness mediator: Neutralize stereotype associations to mitigate bias in large language models. arXiv preprint arXiv:2504.07787, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Safebench: A safety evaluation framework for multimodal large language models

    Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jinyang Guo, Wenbo Zhou, Xianglong Liu, and Dacheng Tao. Safebench: A safety evaluation framework for multimodal large language models. arXiv preprint arXiv:2410.18927, 2024 a

    Pith/arXiv arXiv 2024

  33. [33]

    Jailbreak vision language models via bi-modal adversarial prompt

    Zonghao Ying, Aishan Liu, Tianyuan Zhang, Zhengmin Yu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. Jailbreak vision language models via bi-modal adversarial prompt. arXiv preprint arXiv:2406.04031, 2024 b

    Pith/arXiv arXiv 2024

  34. [34]

    Pushing the limits of safety: A technical report on the atlas challenge 2025

    Zonghao Ying, Siyang Wu, Run Hao, Peng Ying, Shixuan Sun, Pengyu Chen, Junze Chen, Hao Du, Kaiwen Shen, Shangkun Wu, et al. Pushing the limits of safety: A technical report on the atlas challenge 2025. arXiv preprint arXiv:2506.12430, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    From redundancy to relevance: Enhancing explainability in multimodal large language models

    Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Enhancing explainability in multimodal large language models. Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2025

    2025

  36. [36]

    Learning deep features for discriminative localization

    Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2921--2929, 2016

    2016

  37. [37]

    Openpsg: Open-set panoptic scene graph generation via large multimodal models

    Zijian Zhou, Zheng Zhu, Holger Caesar, and Miaojing Shi. Openpsg: Open-set panoptic scene graph generation via large multimodal models. In European Conference on Computer Vision, pp.\ 199--215. Springer, 2024

    2024

  38. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  39. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  40. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  41. [41]

    traffic" and

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2014