Pith. sign in

REVIEW 3 major objections 5 minor 87 references

Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read IEF-VAD claims that synthetic event representations synthesized from ordinary RGB frames, fused with image features by inverse-variance weights, set new state-of-the-art anomaly detection scores on four benchmarks without any event sensor…

desk verdict Solid empirical package, but the Student-t story does not survive inspection: the heavy-tailed correction is a constant rescale that cancels in every fusion weight, so the claimed gains come from the KL loss and test-set hyperparameter selection, not from robust noise modeling. read the letter →

arxiv 2505.02393 v2 pith:B5DANK3F submitted 2025-05-05 cs.CV

Sungheon Jeong , Jihong Park , Mohsen Imani This is my paper · ORCID
classification cs.CV
keywords videoanomalydetectionweaklysupervisedlearningsyntheticeventdatacameramultimodalfusionuncertaintyweightingStudent'stdistributionLaplaceapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most video anomaly detectors see only RGB frames and can miss abrupt, transient motion. This paper proposes IEF-VAD, which synthesizes event-style motion maps directly from RGB videos by thresholding pixel-wise frame differences, encodes them with an event encoder aligned to a pretrained image encoder, and fuses the two modalities by uncertainty-weighted averaging. The central claim is that this fusion overtakes prior weakly supervised detectors on UCF-Crime (AUC 88.67%), XD-Violence (AP 87.63%), ShanghaiTech (AUC 97.98%), and MSAD (AUC 92.90%), with no dedicated event sensor and no frame-level labels. If true, it means an event-camera-like modality can be added to any RGB video dataset, and that a principled inverse-variance weighting prevents the spatially rich image stream from drowning out motion cues.

What carries the argument

The load-bearing machinery is the inverse-variance (precision) weighting rule $w_m = 1/(\tilde{\sigma}_m^2 + \epsilon)$, where $\tilde{\sigma}_m^2$ is the effective variance obtained by Laplace-approximating a Student's t noise model around its mode, giving $\tilde{\sigma}_m^2 = \exp(\log \sigma_m^2 + \log(\nu/(\nu+1)))$. This converts each modality's predicted log-variance into a per-dimension confidence score, and the fused representation is the precision-weighted average of the two modality means. Around that rule sit a Kalman-style sequential update, which merges the previous fused state with the current observation using the same precision weights, and an iterative refinement network that subtracts a fraction $\lambda_r$ of predicted residuals across $N$ steps. The synthetic event maps themselves—binary thresholded differences between consecutive frames (threshold 10/255, clamp 10)—are embedded by an event encoder aligned to the image encoder, which is what makes the motion cues comparable to RGB features.

What would settle it

A decisive check would be to rerun IEF-VAD with the aligned event encoder replaced by a plain linear projection of the raw thresholded difference maps: if AUC/AP stays the same, the event encoder is not the source of the gain. A second check would apply the same fusion to a dataset with real event-camera recordings and see whether the fusion advantage over RGB alone survives.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a heavy-tailed noise model plus a Laplace approximation makes multimodal fusion self-balancing. Each modality's embedding is treated as a noisy observation $z_m = \mu_m + \delta_m$ with Student's t noise; the Laplace approximation converts that noise to an effective Gaussian variance $\tilde{\sigma}_m^2 = \frac{\nu}{\nu+1}\sigma_m^2$, and the fusion weight is the inverse effective variance $w_m = 1/(\tilde{\sigma}_m^2 + \epsilon)$. The fused latent is a precision-weighted average of the image and event means, updated frame by frame in a Kalman-style recursion and then refined by iteratively subtracting predicted residuals. Because the weights are computed per latent dimension from predicted variances, a degraded modality is automatically down-weighted, which the paper demonstrates by masking image features and watching confidence shift to the event stream. Across the four benchmarks the Student's t variant outperforms its Gaussian counterpart and every prior weakly supervised detector.

Load-bearing premise

The load-bearing premise is that synthetic event maps made by thresholding pixel-wise differences between consecutive frames at 10/255 with clamp 10 are faithful enough to real event-camera signals that the aligned event encoder yields genuinely complementary motion cues; the paper's own Limitations section adds that its diagonal-covariance noise model neglects cross-feature correlations and that its regularization weights and degrees of freedom are fixed.

Editorial extensions

If this is right

  • Any RGB video dataset can be augmented with an event-style motion modality through thresholded frame differencing, so multimodal anomaly detection no longer requires owning event cameras.
  • Fusion with value-level inverse-variance weights improves detection on motion-centric classes such as Fighting, Assault, and Shoplifting that RGB alone handles poorly, according to the paper's per-class results.
  • The Student's t noise model with Laplace approximation gives a closed-form KL regularizer and bounded influence of outliers, making the fusion rule more conservative when input is degraded.
  • The method reaches new best published scores on four public benchmarks while using only video-level labels, in line with weakly supervised segment-level training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A decisive check of the paper's framing would be to feed genuine event-camera streams into the same fusion pipeline: if the gains vanish, the 'event modality' is doing the work of a hand-crafted motion feature rather than real event semantics.
  • The thresholded difference maps discard polarity, refractory period, and microsecond timing that real event sensors provide, so the method may understate what a true event modality could contribute; a hardware-validated version of IEF-VAD is a natural next experiment.
  • The per-dimension uncertainty weights behave like an interpretability signal, pointing at the latent dimensions that encode transient motion; the masking study already shows those dimensions react strongly to corruption, which could be mined for explanation tools.
  • If the recipe generalizes, other tasks starved of event data—action recognition, tracking, and surveillance analytics—could adopt the same thresholded-differences-plus-aligned-encoder-plus-precision-fusion template.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes IEF-VAD, a weakly supervised video anomaly detection framework that fuses CLIP image embeddings with synthetic event representations derived from RGB frame differences. The claimed contributions are (i) a Bayesian fusion rule with inverse-variance weights derived from a Student-t noise model via a Laplace approximation, (ii) a Kalman-style sequential update over time, and (iii) an iterative refinement network that denoises the fused latent state. The method is evaluated on UCF-Crime, XD-Violence, ShanghaiTech, and MSAD, where it reports state-of-the-art AUC/AP numbers, with 10-run means and standard deviations. The paper also includes ablations over hyperparameters, loss components, and modality-masking perturbations.

Significance. If the reported results are reproducible and the uncertainty mechanism is honestly characterized, the empirical contribution is useful: the paper demonstrates a practical way to add a motion-centric synthetic event stream to RGB-based weakly supervised anomaly detectors, with consistent gains over unimodal baselines, per-class analysis, and public code. The experiments are run multiple times with standard deviations, which is a strength. However, the paper's central theoretical contribution as written is not sound: as shown in Section 3.2, the Student-t 'effective variance' is a constant rescaling of the Gaussian variance, so the normalized fusion weights and the sequential Kalman-style updates are identical to those of the Gaussian model up to a global scale factor that cancels in every ratio. The reported gains therefore cannot be attributed to heavy-tailed noise modeling, and the paper's framing overstates what the experiments establish. The synthetic-event premise also needs validation because the event encoder is taken from prior work and may not have been trained on thresholded frame-difference maps.

major comments (3)
  1. [Section 3.2, Eq. (3), and Section 3.4, Eq. (5)] The Student-t derivation is vacuous as implemented. Eq. (3) gives sigma_tilde^2 = nu/(nu+1) sigma^2, a constant factor for fixed nu. In the fusion rule mu_f = (w_x mu_x + w_e mu_e)/(w_x + w_e), the weights w_m = 1/(sigma_tilde_m^2 + epsilon) are all scaled by the same constant relative to the Gaussian weights, so the factor cancels exactly (up to the negligible epsilon=1e-8). The sequential update of Section 3.4 has the same property: the state precision is a sum of scaled precisions, and the Kalman-gain ratios are unchanged. Therefore, for fixed predicted log-variances, the Student-t and Gaussian variants produce identical fused means and identical temporal updates. The only place the factor survives is the KL loss L_KL = 1/2(sigma_tilde^2 + mu^2 - 1 - log sigma_tilde^2). Since the variance head predicts log sigma^2, the constant log(nu/(nu+1)) can be absorbed into the bias of h_m, so the model class is identical to a Gaussian model with a different KL regularization weight. The reported improvement from the Student-t variant (e.g., 88.11 to 88.67 on UCF-Crime, Table 1) is therefore attributable to changed KL regularization and network adaptation, not to heavy-tailed sensor noise modeling. This undermines contribution (i) as stated, and the claim in Section 3.2 that the effective variance 'ensures that the fusion process properly accounts for the increased uncertainty due to heavy-tailed noise' is not supported.
  2. [Section G.1, Tables 3 and 4] The final hyperparameters appear to be selected using test-set performance. Tables 3 and 4 report AUC/AP on the evaluation benchmarks for different values of nu, N, epsilon, and lambda_r, and the text describes the best settings as those giving peaks on each dataset (e.g., nu=8, N=10, lambda_r=0.5). The same benchmarks are then used to report the headline results in Table 1. No separate validation split or nested selection protocol is described. This creates a selection bias that is not captured by the reported 10-run standard deviations, and it inflates the claim of state-of-the-art performance. The authors should either use a validation split for model selection or explicitly disclose the selection procedure and re-report results under a protocol that does not tune on the test set.
  3. [Appendix F.1 and Section 3] The validity of the 'event modality' is not established. The paper generates synthetic events by thresholding and clamping pixel-wise differences between consecutive frames (threshold 10/255, clamp 10) and feeds the resulting binary maps into an event encoder [31]. If that encoder was trained on real event camera streams, the synthetic thresholded maps may be out-of-distribution, in which case the method is effectively using a hand-crafted motion feature rather than the event modality the paper claims to integrate. The authors should report whether the encoder of [31] was trained on synthetic or real event data, and ideally include an analysis of the domain gap (e.g., comparing synthetic events with real event data on a small benchmark, or showing that the encoder is robust to this distribution shift). This is load-bearing because the complementarity argument depends on the event stream actually carrying motion cues in the way the paper assumes.
minor comments (5)
  1. [Abstract and Section 1] The abstract contains a typo: 'synthesizes event directly from RGB videos' should read 'synthesizes event representations directly from RGB videos'.
  2. [Section G.1] The parameter epsilon is described as the 'Laplace approximation precision' in the ablation text, but in Eqs. (4)-(5) and the implementation it is a numerical stability constant added to the denominator. This terminology is confusing and should be corrected.
  3. [Table 1 caption and Section 4.1] The caption and text compare IEF-VAD with baselines whose standard deviations are not reported; the conclusion that 'gains are statistically consistent' would be strengthened by reporting baseline variance or significance tests (e.g., paired tests across runs) rather than relying only on the IEF-VAD standard deviations.
  4. [Appendix D] The refinement section in the main text states that the residual is estimated 'based solely on the current fused state,' but Appendix D describes the refinement network as taking 'additional contextual information c_i (which may include time-step context, current effective uncertainty estimates, and modality weights).' These descriptions should be reconciled.
  5. [Appendix F.1] The sentence 'The system had 256GB of system RAM and 5GB VRAM' is internally inconsistent with the stated RTX 6000 Ada 48GB VRAM; this appears to be a typo and should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

Student-t 'effective variance' is a constant rescaling of the Gaussian variance; after normalization it cancels in both static fusion and sequential updates, so the model's heavy-tailed contribution reduces to a KL regularizer.

  1. other [Sec. 3.2, Eq. (3)-(4); Sec. 3.3; Sec. 3.4, Eq. (5); hyperparameter ϵ=10^-8 in App. F.2]
    "This immediately implies ˜σ2 = ν/(ν+1)σ2. Taking the logarithm of both sides gives log ˜σ2 = logσ2 + log(ν/(ν+1)). (3) ... this effective variance is then used in place of the original variance σ2 when computing inverse variance weights ... for the Student’s t model, we use the effective variance: wm = 1/(˜σ2m + ϵ). (4)"

    Because Eq. (3) fixes σ̃² = c σ² with c = ν/(ν+1), the Student-t precision in Eq. (4) is w_t = 1/(c σ² + ϵ) = c⁻¹ w_g (up to ϵ=10⁻⁸), where w_g = 1/(σ² + ϵ) is the Gaussian precision. In the normalized fusion equation µf = (w_x µx + w_e µe)/(w_x + w_e) and in the sequential update of Sec. 3.4, which uses only ratios of these precisions, the factor c cancels exactly for ϵ=0 and is negligible for ϵ=10⁻⁸. Thus, for fixed predicted log-variances, the Student-t and Gaussian variants produce identical fusion weights, identical fused means, and identical temporal updates. The heavy-tailed 'effective variance' therefore does not alter the fusion mechanism; the only place it survives is in L_KL, where it acts as a ν-dependent regularizer.

full rationale

The mathematical reduction is real and checkable from the paper's own equations: σ̃² = ν/(ν+1)σ² is a constant scaling, and every fusion/update formula uses only ratios of inverse variances, so the constant cancels. With ϵ=10⁻⁸ the Student-t and Gaussian variants produce the same fused µ and the same temporal state for fixed network outputs. Thus the paper's first contribution, 'deriving value-level inverse-variance weights' from a heavy-tailed Student-t likelihood, does not change the fusion rule; the only place the scaling survives is L_KL, so the reported Student-t gains are effectively a comparison between two KL regularizers, not evidence for heavy-tailed sensor-noise robustness. This is a partial circularity: the claimed heavy-tailed mechanism is, by construction, equivalent to the Gaussian one for the fusion pathway. I do not find circularity in the use of the authors' event encoder [31], which is an external, code-released component; nor in the synthetic-event preprocessing, which is an empirical validity assumption rather than a definitional loop; nor in the benchmark comparisons, which are self-contained. The empirical SOTA claims remain meaningful as engineering results, but the theoretical attribution is vacuous as implemented.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim relies on learned variance heads, fixed Student-t degrees of freedom, a chosen refinement step count, and hand-chosen event thresholds. The Student-t effective variance is a constant scaling of a learned quantity, so this component introduces no independent model capacity. The event encoder is the authors' prior work.

free parameters (6)
  • nu (Student-t degrees of freedom) = 8
    Selected via test-set ablation on UCF-Crime (Table 3), where nu=8 gives the best AUC; used for all datasets.
  • N (number of refinement steps) = 10
    Selected via ablation; N=0 yields 86.77 AUC vs N=10 at 88.67 AUC on UCF-Crime, so this parameter materially affects the result.
  • lambda_r (refinement attenuation) = 0.5
    Selected via ablation; performance peaks at lambda_r=0.5 in Tables 3 and 4.
  • lambda_1, lambda_2 (regularization weights) = 0.5 each
    Fixed by hand in Appendix F.2 with no sensitivity analysis reported.
  • epsilon (numerical stability) = 1e-8
    Fixed small constant; ablation shows minimal effect on performance.
  • event generation threshold and clamp = 10/255 and 10
    Hand-chosen in preprocessing (Appendix F.1); no sensitivity analysis is reported for these values.
assumptions (5)
  • standard math The Laplace approximation of the Student's t log-density by a Gaussian near the mode is accurate enough for the fusion weights.
    Used to derive the effective variance in Eq. 3 and throughout Section 3.2; the derivation is mathematically correct but the resulting scaling is constant.
  • domain assumption Both modalities zx and ze observe the same underlying scene and share a common spatial structure while exhibiting complementary features.
    Section 3 states this as the basis for fusion; if false, the precision-weighted fusion formula is not motivated.
  • domain assumption Synthetic events produced by simple frame differencing are sufficiently similar to real event data for the CLIP-aligned event encoder [31] to yield useful embeddings.
    Central to the claimed benefit of synthetic events; Appendix F.1 specifies the generation, and the paper does not validate against real event data.
  • domain assumption Weakly supervised segment-level labels, aggregated over 16-frame segments, are sufficient to train the frame-wise classifiers.
    Standard multiple instance learning assumption in video anomaly detection, used in Section 3.6.
  • ad hoc to paper The fixed degrees of freedom nu=8 is appropriate across all datasets.
    Selected via ablation rather than learned or derived; the ablation shows modest sensitivity across datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection." pith.science (2026). https://pith.science/paper/B5DANK3F

@misc{pith2026250502393,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-Weighted Image-Event Multimodal Fusion for Video Anomaly Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B5DANK3F}},
  note         = {Machine review of arXiv:2505.02393}
}
read the original abstract

Most existing video anomaly detectors rely solely on RGB frames, which lack the temporal resolution needed to capture abrupt or transient motion cues, key indicators of anomalous events. To address this limitation, we propose Image-Event Fusion for Video Anomaly Detection (IEF-VAD), a framework that synthesizes event representations directly from RGB videos and fuses them with image features through a principled, uncertainty-aware process. The system (i) models heavy-tailed sensor noise with a Student`s-t likelihood, deriving value-level inverse-variance weights via a Laplace approximation; (ii) applies Kalman-style frame-wise updates to balance modalities over time; and (iii) iteratively refines the fused latent state to erase residual cross-modal noise. Without any dedicated event sensor or frame-level labels, IEF-VAD sets a new state of the art across multiple real-world anomaly detection benchmarks. These findings highlight the utility of synthetic event representations in emphasizing motion cues that are often underrepresented in RGB frames, enabling accurate and robust video understanding across diverse applications without requiring dedicated event sensors. Code and models are available at https://github.com/EavnJeong/IEF-VAD.

Figures

Figures reproduced from arXiv: 2505.02393 by the authors.

Figure 1
Figure 1. Overview of IEF-VAD framework. Each video frame and its corresponding synthetic event representation are processed by CLIP encoders to obtain feature embeddings zm. These are further encoded by modality-specific transformers fm to produce zˆm, which are then passed through projection heads gm and hm to estimate µm and σm. The estimated σm is used to compute the uncertainty-aware fusion weight wm, which is used to ob… view at source ↗
Figure 2
Figure 2. Radar charts showing per-class anomaly detection performance (AUC and AP) for [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Change in image-side uncertainty weights [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 47 canonical work pages

  1. [31]

    Expanding event modality applications through a robust clip-based encoder

    Sungheon Jeong, Hanning Chen, Sanggeon Yun, Suhyeon Cho, Wenjun Huang, Xiangjian Liu, and Mohsen Imani. Expanding event modality applications through a robust clip-based encoder. arXiv preprint arXiv:2412.03093, 2024

    arXiv 2024

  2. [1]

    Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text

    Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in neural information processing systems, 34:24206–24221, 2021

    2021

  3. [2]

    Flamingo: A visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, et al. Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022

    arXiv 2022

  4. [3]

    Synthetic temporal anomaly guided end-to-end video anomaly detection

    Marcella Astrid, Muhammad Zaigham Zaheer, and Seung-Ik Lee. Synthetic temporal anomaly guided end-to-end video anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2021

    2021

  5. [4]

    An application-driven survey on event-based neuromorphic computer vision

    Dario Cazzato and Flavio Bono. An application-driven survey on event-based neuromorphic computer vision. Information, 15(8):472, 2024

    2024

  6. [5]

    Recent event camera innovations: A survey

    Bharatesh Chakravarthi, Aayush Atul Verma, Kostas Daniilidis, Cornelia Fermuller, and Yezhou Yang. Recent event camera innovations: A survey. arXiv preprint arXiv:2408.13627, 2024

    arXiv 2024

  7. [6]

    Prompt-enhanced multiple instance learning for weakly supervised video anomaly detection

    Junxi Chen, Liang Li, Li Su, Zheng-Jun Zha, and Qingming Huang. Prompt-enhanced multiple instance learning for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  8. [7]

    Tevad: Improved video anomaly detection with captions

    Weiling Chen, Keng Teck Ma, Zi Jian Yew, Minhoe Hur, and David Aik-Aun Khoo. Tevad: Improved video anomaly detection with captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5549–5559, 2023

    2023

Show all 87 references
  1. [8]

    Mgfn: Magnitude-contrastive glance-and-focus network for weakly-supervised video anomaly detection

    Yingxian Chen, Zhengzhe Liu, Baoheng Zhang, Wilton Fok, Xiaojuan Qi, and Yik-Chung Wu. Mgfn: Magnitude-contrastive glance-and-focus network for weakly-supervised video anomaly detection. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 387–395, 2023

    2023

  2. [9]

    Label-free event-based object recognition via joint learning with image reconstruction from events

    Hoonhee Cho, Hyeonseong Kim, Yujeong Chae, and Kuk-Jin Yoon. Label-free event-based object recognition via joint learning with image reconstruction from events. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19866–19877, 2023

    2023

  3. [10]

    One model, multiple modalities: A sparsely 10 activated approach for text, sound, image, video and code

    Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. One model, multiple modalities: A sparsely 10 activated approach for text, sound, image, video and code. arXiv preprint arXiv:2205.06126, 2022

    2022 arXiv

  4. [11]

    Laplace redux—effortless bayesian deep learning

    Erik Daxberger et al. Laplace redux—effortless bayesian deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  5. [12]

    Learnable expansion of graph operators for multi-modal feature fusion, 2025

    Dexuan Ding, Lei Wang, Liyun Zhu, Tom Gedeon, and Piotr Koniusz. Learnable expansion of graph operators for multi-modal feature fusion, 2025

    2025

  6. [13]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Siddhartha Srinivasa, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  7. [14]

    Self-supervised video forensics by audio-visual anomaly detection

    Chao Feng, Ziyang Chen, and Andrew Owens. Self-supervised video forensics by audio-visual anomaly detection. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10491–10503, 2023

    2023

  8. [15]

    Mist: Multiple instance self-training framework for video anomaly detection

    Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng. Mist: Multiple instance self-training framework for video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14009–14018, 2021

    2021

  9. [16]

    Convolutional transformer based dual discriminator generative adversarial networks for video anomaly detection

    Xinyang Feng, Dongjin Song, Yuncong Chen, Zhengzhang Chen, Jingchao Ni, and Haifeng Chen. Convolutional transformer based dual discriminator generative adversarial networks for video anomaly detection. arXiv preprint arXiv:2107.13720, 2021

    2021 arXiv

  10. [17]

    Multimodal motion conditioned diffusion model for skeleton- based video anomaly detection

    Alessandro Flaborea, Luca Collorone, Guido Maria D’Amely Di Melendugno, Stefano D’Arrigo, Bardh Prenkaj, and Fabio Galasso. Multimodal motion conditioned diffusion model for skeleton- based video anomaly detection. In Proceedings of the IEEE/CVF international conference on com...

    2023

  11. [18]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning (ICML), 2016

    2016

  12. [19]

    Event- based vision: A survey

    Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event- based vision: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(1):1...

    2020

  13. [20]

    End-to-end learn- ing of representations for asynchronous event-based data

    Daniel Gehrig, Henri Rebecq, Guillermo Gallego, and Davide Scaramuzza. End-to-end learn- ing of representations for asynchronous event-based data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5633–5643, 2019

    2019

  14. [21]

    Anomaly detection in video via self-supervised and multi-task learning

    Mihai Georgescu, Radu Ionescu, et al. Anomaly detection in video via self-supervised and multi-task learning. In CVPR, 2021

    2021

  15. [22]

    Cross-modal fusion and attention mechanism for weakly supervised video anomaly detection

    Ayush Ghadiya, Purbayan Kar, Vishal Chudasama, and Pankaj Wasnik. Cross-modal fusion and attention mechanism for weakly supervised video anomaly detection. arXiv preprint arXiv:2412.20455, 2024

    2024 arXiv

  16. [23]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190, 2023

    2023

  17. [24]

    Omnivore: A single model for many visual modalities

    Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens Van Der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16102–16112, 2022

    2022

  18. [25]

    Multimodal chain-of-thought reasoning in language models

    Zixiang Gong, Shuohang Li, Yuning Shao, et al. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023

    2023 arXiv

  19. [26]

    Deep multimodal representation learning: A survey

    Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey. Ieee Access, 7:63373–63394, 2019

    2019

  20. [27]

    Backprop kf: Learning discriminative deterministic state estimators

    Tuomas Haarnoja, Pieter Abbeel, and Sergey Levine. Backprop kf: Learning discriminative deterministic state estimators. In Advances in Neural Information Processing Systems (NeurIPS), 2016

    2016

  21. [28]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  22. [29]

    Multi-modal sensor fusion for auto driving perception: A survey

    Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, and Yikang Li. Multi-modal sensor fusion for auto driving perception: A survey. arXiv preprint arXiv:2202.02703, 2022. 11

    2022 arXiv

  23. [30]

    Attention-based deep multiple instance learning

    Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127–2136. PMLR, 2018

    2018

  24. [32]

    Multi- modal anomaly detection for unstructured and uncertain environments

    Tianchen Ji, Sri Theja Vuppala, Girish Chowdhary, and Katherine Driggs-Campbell. Multi- modal anomaly detection for unstructured and uncertain environments. arXiv preprint arXiv:2012.08637, 2020

    2012 arXiv

  25. [33]

    What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (NeurIPS), 2017

    Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  26. [34]

    N-imagenet: Towards robust, fine-grained object recognition with event cameras

    Junho Kim, Jaehyeok Bae, Gangin Park, Dongsu Zhang, and Young Min Kim. N-imagenet: Towards robust, fine-grained object recognition with event cameras. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2146–2156, 2021

    2021

  27. [35]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Hexiang Hu, Xiang Shen, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. International Conference on Learning Representations (ICLR), 2023

    2023

  28. [36]

    Align before fuse: Vision and language representation learning with momentum distillation

    Xiang Li, Xiang Zhang, Hang Xu, Zhen Lan, Jing Sun, Jianmin Wang, and Guo-Jun Qi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, 2021

    2021

  29. [37]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024

    2024 arXiv

  30. [38]

    A survey of multimodel large language models

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pages 405–409, 2024

    2024

  31. [39]

    A 128×128 120 db 15µs latency asynchronous temporal contrast vision sensor

    Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128×128 120 db 15µs latency asynchronous temporal contrast vision sensor. IEEE journal of solid-state circuits, 43(2):566– 576, 2008

    2008

  32. [40]

    Future frame prediction for anomaly detection–a new baseline

    Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection–a new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6536–6545, 2018

    2018

  33. [41]

    Towards multimodal model generalization: Visual-audio temporal alignment for video classification

    Yixuan Liu, Linchao Zhang, Yabiao Wang, Ying Wang, Gang Wang, and Hongsheng Li. Towards multimodal model generalization: Visual-audio temporal alignment for video classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10970–...

    2021

  34. [42]

    Weakly supervised temporal anomaly localization in surveillance videos

    Yuchen Liu, Yumin Tian, Yanning Zhang, and Kai Chen. Weakly supervised temporal anomaly localization in surveillance videos. In ACM MM, 2021

    2021

  35. [43]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  36. [44]

    Learning optical flow from event camera with rendered dataset

    Xinglong Luo, Kunming Luo, Ao Luo, Zhengning Wang, Ping Tan, and Shuaicheng Liu. Learning optical flow from event camera with rendered dataset. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9847–9857, 2023

    2023

  37. [45]

    Video anomaly detection and explanation via large language models

    Hui Lv and Qianru Sun. Video anomaly detection and explanation via large language models. arXiv preprint arXiv:2401.05702, 2024

    2024 arXiv

  38. [46]

    Unbiased multiple instance learning for weakly supervised video anomaly detection

    Hui Lv, Zhongqi Yue, Qianru Sun, Bin Luo, Zhen Cui, and Hanwang Zhang. Unbiased multiple instance learning for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8022–8031, 2023

    2023

  39. [47]

    Fusion framework and multimodality for the laplacian approximation of bayesian neural networks

    Magnus Malmström, Isaac Skog, Daniel Axehill, and Fredrik Gustafsson. Fusion framework and multimodality for the laplacian approximation of bayesian neural networks. arXiv preprint arXiv:2310.08315, 2023

    2023 arXiv

  40. [48]

    Multimodal deep learning

    Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In ICML, pages 689–696, 2011. 12

    2011

  41. [49]

    The promises and pitfalls of bayesian deep learning in computer vision

    Sebastian Ober, Christoph Posch, Max Welling, Yarin Gal, and Fabio Cuzzolin. The promises and pitfalls of bayesian deep learning in computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  42. [50]

    Converting static image datasets to spiking neuromorphic datasets using saccades

    Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015

    2015

  43. [51]

    Can you trust your model’s uncertainty? evaluating predictive uncer- tainty under dataset shift

    Yaniv Ovadia and et al. Can you trust your model’s uncertainty? evaluating predictive uncer- tainty under dataset shift. In NeurIPS, 2019

    2019

  44. [52]

    Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy

    Federico Paredes-Vallés and Guido CHE De Croon. Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3446–3455, 2021

    2021

  45. [53]

    Assessing modality bias in video question answering benchmarks with multimodal large language models

    Kyunghyun Park et al. Assessing modality bias in video question answering benchmarks with multimodal large language models. arXiv preprint arXiv:2408.12763, 2024

    2024 arXiv

  46. [54]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  47. [55]

    Events-to-video: Bringing modern computer vision to event cameras

    Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to-video: Bringing modern computer vision to event cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3857–3866, 2019

    2019

  48. [56]

    Multimodal foundation models: From specialists to general-purpose learners

    Sheng Shen, Shijie Tang, Di Niu, and et al. Multimodal foundation models: From specialists to general-purpose learners. arXiv preprint arXiv:2306.05425, 2023

    2023 arXiv

  49. [57]

    Secrets of event-based optical flow

    Shintaro Shiba, Yoshimitsu Aoki, and Guillermo Gallego. Secrets of event-based optical flow. In European Conference on Computer Vision, pages 628–645. Springer, 2022

    2022

  50. [58]

    Uncertainty aware audiovisual activity recognition using deep bayesian variational inference

    Mahesh Subedar, Ranganath Krishnan, Paulo Lopez Meyer, Omesh Tickoo, and Jonathan Huang. Uncertainty aware audiovisual activity recognition using deep bayesian variational inference. arXiv preprint arXiv:1811.10811, 2018

    2018 arXiv

  51. [59]

    Real-world anomaly detection in surveillance videos

    Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6479–6488, 2018

    2018

  52. [60]

    Hawk: Learning to understand open-world video anomalies

    Jiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, and Yingcong Chen. Hawk: Learning to understand open-world video anomalies. Advances in Neural Information Processing Systems, 37:139751–139785, 2024

    2024

  53. [61]

    Weakly-supervised video anomaly detection with robust temporal feature magnitude learning

    Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W Verjans, and Gustavo Carneiro. Weakly-supervised video anomaly detection with robust temporal feature magnitude learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4975–4986, 2021

    2021

  54. [62]

    Multimodal transformer for unaligned multimodal language sequences

    Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558–6569, 2019

    2019

  55. [63]

    Exploiting spatial sparsity for event cameras with visual transformers

    Zuowen Wang, Yuhuang Hu, and Shih-Chii Liu. Exploiting spatial sparsity for event cameras with visual transformers. In 2022 IEEE International Conference on Image Processing (ICIP), pages 411–415. IEEE, 2022

    2022

  56. [64]

    An introduction to the kalman filter.University of North Carolina at Chapel Hill, Department of Computer Science, 7(1):1–16, 1995

    Greg Welch and Gary Bishop. An introduction to the kalman filter.University of North Carolina at Chapel Hill, Department of Computer Science, 7(1):1–16, 1995

    1995

  57. [65]

    Student-t processes for bayesian deep learning

    Jianxiang Wu, Kailun Ren, Yisen Wang, Xiaotong Liu, and Jinfeng Wang. Student-t processes for bayesian deep learning. In International Conference on Learning Representations (ICLR), 2021

    2021

  58. [66]

    Not only look, but also listen: Learning multimodal violence detection under weak supervision

    Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedi...

    2020

  59. [67]

    Weakly supervised audio-visual violence detection

    Peng Wu, Xiaotao Liu, and Jing Liu. Weakly supervised audio-visual violence detection. IEEE Transactions on Multimedia, 25:1674–1685, 2022

    2022

  60. [68]

    Open-vocabulary video anomaly detection

    Peng Wu, Xuerong Zhou, Guansong Pang, Yujia Sun, Jing Liu, Peng Wang, and Yanning Zhang. Open-vocabulary video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18297–18307, 2024

    2024

  61. [69]

    Weakly supervised video anomaly detection and localization with spatio- temporal prompts

    Peng Wu, Xuerong Zhou, Guansong Pang, Zhiwei Yang, Qingsen Yan, Peng Wang, and Yanning Zhang. Weakly supervised video anomaly detection and localization with spatio- temporal prompts. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 9301–9310, 2024

    2024

  62. [70]

    Vadclip: Adapting vision-language models for weakly supervised video anomaly detection

    Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, and Yanning Zhang. Vadclip: Adapting vision-language models for weakly supervised video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6074–6082, 2024

    2024

  63. [71]

    Eventclip: Adapting clip for event-based object recognition

    Ziyi Wu, Xudong Liu, and Igor Gilitschenski. Eventclip: Adapting clip for event-based object recognition. arXiv preprint arXiv:2306.06354, 2023

    2023 arXiv

  64. [72]

    Multimodal learning with transformers: A survey

    Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12113–12132, 2023

    2023

  65. [73]

    Event camera data pre-training

    Yan Yang, Liyuan Pan, and Liu Liu. Event camera data pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10699–10709, 2023

    2023

  66. [74]

    Explainable video anomaly detection via verbalized learning of vision-language models

    Muchao Ye, Weiyang Liu, and Pan He. Explainable video anomaly detection via verbalized learning of vision-language models. arXiv preprint arXiv:2412.01095, 2024

    2024 arXiv

  67. [75]

    Harness- ing large language models for training-free video anomaly detection

    Luca Zanella, Willi Menapace, Massimiliano Mancini, Yiming Wang, and Elisa Ricci. Harness- ing large language models for training-free video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  68. [76]

    Holmes-vau: Towards long-term video anomaly understanding at any granularity

    Huaxin Zhang, Xiaohao Xu, Xiang Wang, Jialong Zuo, Xiaonan Huang, Changxin Gao, Shanjun Zhang, Li Yu, and Nong Sang. Holmes-vau: Towards long-term video anomaly understanding at any granularity. arXiv preprint arXiv:2412.06171, 2024

    2024 arXiv

  69. [77]

    Event-based vision: A survey

    Yuhuang Zhang, Guillermo Gallego, Davide Scaramuzza, et al. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):154–180, 2022

    2022

  70. [78]

    Mm-react: Prompting chatgpt for multimodal reasoning and action

    Wayne Zhao, Xisen Wang, and et al. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023

    2023 arXiv

  71. [79]

    Deep learning for event-based vision: A comprehensive survey and benchmarks

    Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang. Deep learning for event-based vision: A comprehensive survey and benchmarks. arXiv preprint arXiv:2302.08890, 2023

    2023 arXiv

  72. [80]

    Gtad: A semi-supervised learning framework for temporal anomaly detection in surveillance videos

    Zheng Zhong, Wenhao Li, Xiaojun Zhao, et al. Gtad: A semi-supervised learning framework for temporal anomaly detection in surveillance videos. In ECCV, 2022

    2022

  73. [81]

    Dual memory units with uncertainty regulation for weakly supervised video anomaly detection

    Hang Zhou, Junqing Yu, and Wei Yang. Dual memory units with uncertainty regulation for weakly supervised video anomaly detection. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3769–3777, 2023

    2023

  74. [82]

    Eventbind: Learning a unified repre- sentation to bind them all for event-based open-world understanding

    Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, and Lin Wang. Eventbind: Learning a unified repre- sentation to bind them all for event-based open-world understanding. In European Conference on Computer Vision, pages 477–494. Springer, 2024

    2024

  75. [83]

    Anomalynet: An anomaly detection network for video surveillance

    Yao Zhou, Xiaodong Liu, Yadong Wang, and Weiming Wang. Anomalynet: An anomaly detection network for video surveillance. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1700–1704, 2019

    2019

  76. [84]

    The multivehicle stereo event camera dataset: An event camera dataset for 3d perception

    Alex Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. The multivehicle stereo event camera dataset: An event camera dataset for 3d perception. In IEEE RA-L, 2018

    2018

  77. [85]

    A variational bayesian approach to robust sensor fusion based on student-t distribution

    Hao Zhu, Henry Leung, and Zhongshi He. A variational bayesian approach to robust sensor fusion based on student-t distribution. Information Sciences, 221:201–214, 2013

    2013

  78. [86]

    Advancing video anomaly detection: A concise review and a new dataset

    Liyun Zhu, Lei Wang, Arjun Raj, Tom Gedeon, and Chen Chen. Advancing video anomaly detection: A concise review and a new dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. 14

    2024

  79. [87]

    synthetic

    Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-supervised multimodal learning: A survey. arXiv preprint arXiv:2304.01008, 2023. 15 A Bounded Influence of Student’s t-Noise Proposition 1 (Robustness of Student’s t to Outliers). Letδ be a noise or residual term dra...

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.