Pith. sign in

REVIEW 4 major objections 9 minor 1 cited by

Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning

T0 review · 4 major / 9 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multimodal spiking neural networks can overcome modality imbalance and temporal misalignment by scoring every timestep of the fused spike train, reporting 77.55%, 70.65%, and 97.5% accuracy on CREMA-D, AVE, and EAD with 3-5 timesteps.

desk verdict A plausible but sloppily specified temporal-attention mechanism for balanced multimodal SNNs; the ablation undercuts the time-alignment story, and the key equation needs fixing before the results can be trusted. read the letter →

arxiv 2505.14535 v1 pith:BGMNHUMD submitted 2025-05-20 cs.LG cs.HC

Jiangrong Shen , Yulin Xie , Qi Xu , Gang Pan , Huajin Tang , Badong Chen This is my paper · ORCID
classification cs.LGcs.HC
keywords spikingneuralnetworksmultimodallearningtemporalattentionmodalityimbalanceadaptivefusionlossaudio-visualclassificationenergyefficiencyneuromorphiccomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the failure mode of multimodal spiking neural networks is temporal, not just architectural: when audio and visual spike trains are averaged or statically weighted, the faster-converging modality dominates and the timing of informative events is lost. It proposes the temporal attention-guided adaptive fusion (TAAF) module, which scores every timestep of the fused spike train, and a temporal adaptive balanced fusion loss that uses those scores to slow down the dominant modality. The reported results are 77.55% accuracy on CREMA-D, 70.65% on AVE, and 97.5% on EAD, all above the ANN and SNN baselines listed in the paper, at only 3-5 timesteps and with a lower estimated energy cost than a comparable ANN. A sympathetic reader would care because it suggests that modeling the temporal dimension of fusion is a concrete lever for making energy-efficient neuromorphic systems competitive on real audio-visual tasks.

What carries the argument

The load-bearing object is the temporal attention score $\alpha_u(t)$, averaged from the softmax similarity matrix $A(Q_u, K_u)$ of query-key projections of each modality's output. It is used twice: first to reweight loss contributions per timestep, and second, multiplied by the modality contribution ratio $\rho_u$, to form the per-modality suppression factor $k_u = 1 - \tanh(\alpha_u \rho_u)$ in the final balanced loss. A convolutional time-alignment module, adapted from the balanced multimodal SNN work cited as [50], first synchronizes unimodal spike trains so the attention scores are computed on temporally aligned features.

What would settle it

Run the full model with the temporal attention scores replaced by uniform weights at every timestep, keeping the time-alignment and balanced-loss components unchanged; if CREMA-D accuracy does not drop materially below the reported 77.55%, the attention scores are not doing the claimed work.

Watch

Extended reading notes

Core claim

The central claim is that assigning an importance score to each timestep of the fused spiking representation, and using that score to reweight both the features and the loss, resolves modality imbalance that static fusion leaves untouched. The TAAF module computes temporal attention scores $\alpha_u(t)$ by averaging a scaled dot-product similarity matrix over timesteps; these scores modulate the temporal cross-entropy loss, while true-class softmax probabilities for each modality form a contribution ratio $\rho_u$ that yields the suppression factor $k_u = 1 - \tanh(\alpha_u \rho_u)$ in the final loss. The paper reports that this mechanism improves CREMA-D accuracy to 77.55%, AVE accuracy to 70.65%, and EAD accuracy to 97.5%, and that epoch-wise trajectories show the dominant audio branch being rebalanced as visual updates are strengthened around epoch 55.

Load-bearing premise

The framework assumes its internal importance scores and per-modality probability estimates genuinely reflect which moments and which senses carry the task, even though nothing stops the model from shrinking the influence of hard timesteps just to lower its loss.

Editorial extensions

If this is right

  • The reported operation at 3-5 timesteps, against SNN baselines that use up to 20 timesteps, implies that temporal attention can extract the task-relevant moments earlier without sacrificing accuracy.
  • The small accuracy gap between concatenation and summation fusion in the full model (77.55% vs 76.75% on CREMA-D) implies temporal attention makes fusion quality less dependent on the fusion operator, which matters for neuromorphic hardware where concatenation is costly.
  • The loss modulation changes training dynamics: on CREMA-D the visual branch overtakes the audio branch around epoch 55 and fusion accuracy jumps near epoch 75, consistent with the claim that the mechanism actively counters modality dominance.
  • The energy estimate of 18.5 MJ for the proposed SNN versus 49.91 MJ for a comparable ANN on CREMA-D implies that the reported accuracy gains do not have to be bought with extra energy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the learned attention scores are semantically interpretable; if they are, a natural extension is to stop simulation at low-importance timesteps at inference time, saving additional energy beyond the reported estimate. This is an editorial inference.
  • Because the modality contribution ratio is built from true-class softmax probabilities, the balancing mechanism could be misled by a confidently wrong modality; an alternative estimator based on validation accuracy or gradient norms would be a direct test of whether this part of the loss is the right carrier of the effect. This is an editorial inference.
  • The same per-timestep reweighting could be applied to ANN multimodal models to separate the contribution of temporal attention from the contribution of spike-based dynamics; the paper evaluates only SNNs, so that comparison remains open. This is an editorial inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. The manuscript proposes a temporal attention-guided adaptive fusion (TAAF) framework for multimodal spiking neural networks, together with a temporal adaptive balanced fusion loss. The TAAF module is intended to compute per-timestep importance scores over fused unimodal spiking features; these scores are then used both to reweight the cross-entropy loss over timesteps and to scale unimodal loss terms through modality contribution ratios. Experiments on CREMA-D, AVE, and EAD report state-of-the-art accuracies of 77.55%, 70.65%, and 97.5%, respectively, with an estimated energy advantage over a comparable ANN. The central claimed contribution is that temporally fine-grained attention resolves modality imbalance and temporal misalignment in SNNs.

Significance. If the claims were established, the paper would make a useful contribution to energy-efficient multimodal SNN training: per-timestep attention-guided fusion is a natural and timely idea, and the experimental scope covers three datasets with multiple fusion strategies. I also credit the paper for comparing against recent ANN and SNN baselines and for attempting ablation and convergence analyses. However, the defining equations do not currently deliver a per-timestep attention vector, the headline ablation contains a configuration in which the proposed mechanism degrades accuracy, and the energy claim is not accompanied by a measurement protocol; the significance is therefore conditional on substantial revision. No code or data release is indicated, which further limits verification.

major comments (4)
  1. [§3.1, Eqs. (5)–(8), and §3.2] The central object α_u is defined in Eq. (7) as α_u = (1/T) Σ_{i=1}^T A_i(Q_u,K_u), which is a scalar per modality, but Eq. (8) uses α_u(t) as a per-timestep weight inside a sum over t, and §3.2 defines k_t^u = 1 − tanh(α_u ρ_t^u). No equation defines the length-T vector α_u(t); A_i(Q_u,K_u) indexes rows of the T×T similarity matrix and is not a temporal slice indexed by the same t as the output O_u(t). If α_u is a scalar, L_AGL_ori in Eq. (8) is ordinary cross-entropy scaled by a constant and the 'temporal adaptive' part of the loss is vacuous; if a vector is intended, Eq. (7) must be corrected and the shapes of W_Q^u and W_K^u relative to O_u must be specified. Because every downstream modulation and the claimed temporal attention mechanism depend on this quantity, this is a load-bearing inconsistency.
  2. [Table 2 and §4.2] Table 2, CREMA-D summation row: 'TAAF-SNNs without TA' attains 79.17 while the full TAAF-SNNs model attains 76.75, so removing the component denoted TA improves accuracy by 2.42 points under summation fusion. The text in §4.2 does not discuss this contrary result. Moreover, the claimed '1.8% accuracy gain on CREMA-D (77.55% vs. 75.67%)' compares the concatenation row of the full model with the summation row of 'SNN with LA and TA', i.e., it mixes fusion strategies and does not isolate the attention mechanism. With single-run numbers and no significance tests, these results do not support the paper's central claim that temporal attention or time alignment improves multimodal SNN performance.
  3. [§4.3, energy estimate] The sentence 'The estimated energy consumption of the proposed TAAF-SNNs and the ANNs with the similar architecture ... is 18.5 MJ and 49.91 MJ' provides no methodology. The estimate omits synaptic operation counting, spike rate assumptions, timestep counts, hardware parameters, and the network configuration compared; without this information the energy-efficiency claim cannot be verified, reproduced, or compared against the reported ANN baseline.
  4. [§4, Tables 1 and 2] All accuracy numbers are single point estimates without standard deviations, number of seeds, or statistical tests. The reported gains are often small (e.g., 70.65 vs. 68.04 on AVE; 77.55 vs. 77.14 on CREMA-D), and the CREMA-D ablation in Table 2 shows a reversal. Repeated runs and significance reporting are necessary before any state-of-the-art claim can be made.
minor comments (9)
  1. [Keywords] The keyword 'Muti-modal' should be 'Multi-modal'.
  2. [Global] The text frequently prints 'A VE' with an extra space (e.g., Sections 3 and 4); this should be 'AVE'.
  3. [Abstract vs. §4.1] The abstract reports 97.5% accuracy on EAD while Section 4.1 reports 97.59%; the two numbers should be reconciled.
  4. [Table 1, AVE block] The same model 'MISNET-XL [18]' appears twice with identical fusion strategy and differing accuracies 67.24 and 68.04; the entries should be distinguished by configuration or timestep, or corrected.
  5. [Algorithm 1, line 8] The line 'Compute losses L_m2, L_m2, L_f' should read 'L_m1, L_m2, L_f'.
  6. [Figure 2 caption] The fourth panel is labeled '(4)' in the caption but should be '(d)'.
  7. [Eq. (10)] The notation k_m1, k_m2 is introduced only as k_t^u with a time index; Eq. (10) omits the time index, so it should state whether the per-timestep factors are averaged or applied per timestep.
  8. [§3.1, dimensionality] The dimensions of O_u and the projection matrices are not fully specified; if O_u is a T-step sequence, the shapes in Eq. (5) need to make the time index explicit, which is related to the main technical point above.
  9. [§4, experimental setup] No details are given for preprocessing, spike encoding, number of runs, hardware, or hyperparameter search; a reproducibility section would be needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attention-modulated loss is a training objective, not a derived prediction, and the reported accuracies are benchmarked externally. The self-cited time-alignment module is a normal method citation rather than a load-bearing circular reduction.

full rationale

The central claimed mechanism is the TAAF module and the temporal adaptive balanced fusion loss. The attention scores are computed from the network's own outputs (Eqs. 5-7) and then used to reweight those same outputs in Eq. 8 and to form the modality factors in Eq. 10. This is self-referential in an engineering sense, but it is a training objective rather than a first-principles derivation of an external quantity. No fitted parameter is later renamed as a prediction, and the accuracy claims are measured on CREMA-D, AVE, and EAD against published baselines. The only potentially self-citation-related element is the time-alignment module, which is adapted from [50] with further details deferred to that citation; [50] is a prior paper with overlapping authors, so this is a self-citation. However, it is a normal method citation to a separate peer-reviewed work and does not by itself make the TAAF contribution or the benchmark results circular. The internal shape ambiguity of Eq. 7 (whether alpha_u is a scalar average or a length-T vector) and the Table 2 ablation anomaly are correctness and consistency risks, not circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical architecture and loss improvement, so the ledger is mostly hyperparameters and modeling choices rather than derived constants or invented entities. No new physical entities are postulated. The main burden is the set of ad hoc modeling assumptions around temporal attention and modality-dominance measurement.

free parameters (7)
  • lambda (loss interpolation weight in Eq. 9) = not reported
    Balances attention-weighted cross-entropy against the membrane-potential MSE term. Chosen by hand; no sensitivity analysis.
  • beta (unimodal loss weight in Eq. 10) = not reported
    Controls the strength of the unimodal loss terms relative to the fused loss.
  • Phi (membrane potential normalization hyperparameter) = not reported
    Used in L_MSE to normalize membrane potentials; no value or search range is given.
  • tau (LIF leakage coefficient) = 0.5
    Fixed leakage coefficient in Eq. 1; standard but hand-chosen.
  • per-dataset timestep counts = 3 (CREMA-D), 4 or 3 (AVE), 5 (EAD)
    Timestep counts are selected per dataset; lower counts are claimed to help energy efficiency and temporal focus.
  • modulation end epoch = 70 (AVE)
    Hyperparameter that stops the per-modality learning-rate modulation; the paper states this is to guarantee final convergence.
  • attention projection matrices W_Q and W_K = learned end-to-end
    These learned matrices produce the temporal attention scores that drive both fusion and loss modulation; their values come from training data.
assumptions (5)
  • domain assumption Leaky integrate-and-fire dynamics with fixed leakage tau=0.5 and hard reset (Eqs. 1-3) are a valid model of the feature extractor.
    The backbone is a spiking ResNet18 with LIF neurons; the results depend on this simulation model, which is standard but not derived here.
  • ad hoc to paper Scaled dot-product softmax attention over timesteps captures cross-modal temporal importance.
    Eqs. 5-7 define temporal attention as softmax(QK^T/sqrt(C)) averaged over rows; there is no external validation that this score measures task-relevant temporal moments.
  • ad hoc to paper The classifier can be decomposed into two half classifiers with W=[W_m1,W_m2] so that unimodal outputs faithfully estimate each modality's contribution.
    Section 3.1 splits the fused classifier to simulate unimodal outputs; this decomposition is an architectural assumption, not a proved equivalence.
  • ad hoc to paper Softmax true-class probability of each unimodal output is a reliable measure of modality dominance.
    The contribution ratios rho_t_m1 and rho_t_m2 in Section 3.2 are built from these probabilities; early in training these probabilities are noisy.
  • domain assumption The convolutional time-alignment module from [50] synchronizes spike trains without losing discriminative information.
    The framework relies on this module to unify temporal scales, yet Table 2 shows removing it improves accuracy in one configuration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning." pith.science (2026). https://pith.science/paper/BGMNHUMD

@misc{pith2026250514535,
  author       = {Pith},
  title        = {Pith review of: Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGMNHUMD}},
  note         = {Machine review of arXiv:2505.14535}
}
read the original abstract

Multimodal spiking neural networks (SNNs) hold significant potential for energy-efficient sensory processing but face critical challenges in modality imbalance and temporal misalignment. Current approaches suffer from uncoordinated convergence speeds across modalities and static fusion mechanisms that ignore time-varying cross-modal interactions. We propose the temporal attention-guided adaptive fusion framework for multimodal SNNs with two synergistic innovations: 1) The Temporal Attention-guided Adaptive Fusion (TAAF) module that dynamically assigns importance scores to fused spiking features at each timestep, enabling hierarchical integration of temporally heterogeneous spike-based features; 2) The temporal adaptive balanced fusion loss that modulates learning rates per modality based on the above attention scores, preventing dominant modalities from monopolizing optimization. The proposed framework implements adaptive fusion, especially in the temporal dimension, and alleviates the modality imbalance during multimodal learning, mimicking cortical multisensory integration principles. Evaluations on CREMA-D, AVE, and EAD datasets demonstrate state-of-the-art performance (77.55\%, 70.65\% and 97.5\%accuracy, respectively) with energy efficiency. The system resolves temporal misalignment through learnable time-warping operations and faster modality convergence coordination than baseline SNNs. This work establishes a new paradigm for temporally coherent multimodal learning in neuromorphic systems, bridging the gap between biological sensory processing and efficient machine intelligence.

Figures

Figures reproduced from arXiv: 2505.14535 by the authors.

Figure 1
Figure 1. Architecture of the proposed attention-guided multimodal SNN. The temporal attention-guided adaptive [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The epoch-wise analysis of accuracy and loss trajectories with the proposed multimodal SNNs on CREMA-D [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SETransformer: A Hybrid Attention-Based Architecture for Robust Human Activity Recognition

    cs.LG 2025-05 reject novelty 2.0 of 10

    SETransformer combines a Transformer encoder, channel attention, and attention pooling for WISDM activity recognition, but the architecture is permutation-invariant and the reported comparison omits the model itself.

Reference graph

Works this paper leans on

61 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [50]

    The balanced multi-modal spiking neural networks with online loss adjustment and time alignment

    Jianing Han, Jiangrong Shen, Qi Xu, Jian Liu, and Huajin Tang. The balanced multi-modal spiking neural networks with online loss adjustment and time alignment. In2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2024

    work page 2024

  2. [1]

    A multi-modal parcellation of human cerebral cortex.Nature, 536(7615):171–178, 2016

    Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex.Nature, 536(7615):171–178, 2016

    2016

  3. [2]

    Cortical tracking of hierarchical rhythms orchestrates the multisensory processing of biological motion.eLife, 13:RP98701, 2025

    Li Shen, Shuo Li, Yuhao Tian, Ying Wang, and Yi Jiang. Cortical tracking of hierarchical rhythms orchestrates the multisensory processing of biological motion.eLife, 13:RP98701, 2025

    work page 2025

  4. [3]

    Cmci: A robust multimodal fusion method for spiking neural networks

    Runhao Jiang, Jianing Han, Yingying Xue, Ping Wang, and Huajin Tang. Cmci: A robust multimodal fusion method for spiking neural networks. InInternational Conference on Neural Information Processing, pages 159–171. Springer, 2023

    work page 2023

  5. [4]

    Event-based multimodal spiking neural network with attention mechanism

    Qianhui Liu, Dong Xing, Lang Feng, Huajin Tang, and Gang Pan. Event-based multimodal spiking neural network with attention mechanism. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8922–8926. IEEE, 2022

    work page 2022

  6. [5]

    Balanced multimodal learning via on-the-fly gradient modulation

    Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8238–8247, 2022

    2022

  7. [6]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. InProceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040–13051, 2024

    work page 2024

  8. [7]

    Networks of spiking neurons: The third generation of neural network models.Neural Networks, 10(9):1659–1671, 1997

    Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models.Neural Networks, 10(9):1659–1671, 1997. Number: 9

    work page 1997

Show all 61 references
  1. [8]

    A biomimetic neural encoder for spiking neural network.Nature communications, 12(1):2143, 2021

    Shiva Subbulakshmi Radhakrishnan, Amritanand Sebastian, Aaryan Oberoi, Sarbashis Das, and Saptarshi Das. A biomimetic neural encoder for spiking neural network.Nature communications, 12(1):2143, 2021

    2021

  2. [9]

    Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence.Science Advances, 9(40):eadi1480, 2023

    Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence.Science Advances, 9(40):eadi1480, 2023

    2023

  3. [10]

    Qi Xu, Yuyuan Gao, Jiangrong Shen, Yaxin Li, Xuming Ran, Huajin Tang, and Gang Pan. Enhancing adaptive history reserving by spiking convolutional block attention module in recurrent neural networks.Advances in Neural Information Processing Systems, 36:58890–58901, 2023

    2023

  4. [11]

    Efficient spiking neural networks with sparse selective activation for continual learning

    Jiangrong Shen, Wenyao Ni, Qi Xu, and Huajin Tang. Efficient spiking neural networks with sparse selective activation for continual learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 611–619, 2024

    2024

  5. [12]

    State transition of dendritic spines improves learning of sparse spiking neural networks

    Yanqi Chen, Zhaofei Yu, Wei Fang, Zhengyu Ma, Tiejun Huang, and Yonghong Tian. State transition of dendritic spines improves learning of sparse spiking neural networks. InInternational Conference on Machine Learning, pages 3701–3715. PMLR, 2022

    2022

  6. [13]

    Computational event-driven vision sensors for in-sensor spiking neural networks.Nature Electronics, 6(11):870–878, 2023

    Yue Zhou, Jiawei Fu, Zirui Chen, Fuwei Zhuge, Yasai Wang, Jianmin Yan, Sijie Ma, Lin Xu, Huanmei Yuan, Mansun Chan, et al. Computational event-driven vision sensors for in-sensor spiking neural networks.Nature Electronics, 6(11):870–878, 2023

    2023

  7. [14]

    Towards energy efficient spiking neural networks: An unstructured pruning framework

    Xinyu Shi, Jianhao Ding, Zecheng Hao, and Zhaofei Yu. Towards energy efficient spiking neural networks: An unstructured pruning framework. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  8. [15]

    Reversing structural pattern learning with biologically inspired knowledge distillation for spiking neural networks

    Qi Xu, Yaxin Li, Xuanye Fang, Jiangrong Shen, Qiang Zhang, and Gang Pan. Reversing structural pattern learning with biologically inspired knowledge distillation for spiking neural networks. InProceedings of the 32nd ACM International Conference on Multimedia, pages 3431–3439, 2024

    2024

  9. [16]

    Improving the sparse structure learning of spiking neural networks from the view of compression efficiency.arXiv preprint arXiv:2502.13572, 2025

    Jiangrong Shen, Qi Xu, Gang Pan, and Badong Chen. Improving the sparse structure learning of spiking neural networks from the view of compression efficiency.arXiv preprint arXiv:2502.13572, 2025. 11 APREPRINT- SEPTEMBER18, 2025

    2025 arXiv

  10. [17]

    Nitin Rathi and Kaushik Roy. Stdp based unsupervised multimodal learning with cross-modal processing in spiking neural networks.IEEE Transactions on Emerging Topics in Computational Intelligence, 5(1):143–153, 2018

    2018

  11. [18]

    Towards energy-efficient audio-visual classification via multimodal interactive spiking neural network.ACM Transactions on Multimedia Computing, Communications and Applications

    Xu Liu, Na Xia, Jinxing Zhou, Zhangbin Li, and Dan Guo. Towards energy-efficient audio-visual classification via multimodal interactive spiking neural network.ACM Transactions on Multimedia Computing, Communications and Applications

  12. [19]

    Digit recognition using multimodal spiking neural networks.arXiv preprint arXiv:2409.00552, 2024

    William Bjorndahl, Jack Easton, Austin Modoff, Eric C Larson, Joseph Camp, and Prasanna Rangarajan. Digit recognition using multimodal spiking neural networks.arXiv preprint arXiv:2409.00552, 2024

    2024 arXiv

  13. [20]

    Transformer- based spiking neural networks for multimodal audiovisual classification.IEEE Transactions on Cognitive and Developmental Systems, 16(3):1077–1086, 2023

    Lingyue Guo, Zeyu Gao, Jinye Qu, Suiwu Zheng, Runhao Jiang, Yanfeng Lu, and Hong Qiao. Transformer- based spiking neural networks for multimodal audiovisual classification.IEEE Transactions on Cognitive and Developmental Systems, 16(3):1077–1086, 2023

    2023

  14. [21]

    Fusionsense: Emotion classification using feature fusion of multimodal data and deep learning in a brain-inspired spiking neural network.Sensors, 20(18):5328, 2020

    Clarence Tan, Gerardo Ceballos, Nikola Kasabov, and Narayan Puthanmadam Subramaniyam. Fusionsense: Emotion classification using feature fusion of multimodal data and deep learning in a brain-inspired spiking neural network.Sensors, 20(18):5328, 2020

    2020

  15. [22]

    Enhancing robustness against adversarial attacks in multimodal emotion recognition with spiking transformers.IEEE Access, 2025

    Guoming Chen, Zhuoxian Qian, Dong Zhang, Shuang Qiu, and Ruqi Zhou. Enhancing robustness against adversarial attacks in multimodal emotion recognition with spiking transformers.IEEE Access, 2025

    2025

  16. [23]

    Smile: Spiking multi-modal interactive label- guided enhancement network for emotion recognition

    Ming Guo, Wenrui Li, Chao Wang, Yuxin Ge, and Chongjun Wang. Smile: Spiking multi-modal interactive label- guided enhancement network for emotion recognition. In2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2024

    2024

  17. [24]

    Brain-like evolving spiking neural networks for multimodal information processing

    Simei Gomes Wysoski, Lubica Benuskova, and Nikola Kasabov. Brain-like evolving spiking neural networks for multimodal information processing. InBrain-Inspired Information Technology, pages 15–27. Springer, 2010

    2010

  18. [25]

    Enhancing snn-based spatio-temporal learning: A benchmark dataset and cross-modality attention model.Neural Networks, 180:106677, 2024

    Shibo Zhou, Bo Yang, Mengwen Yuan, Runhao Jiang, Rui Yan, Gang Pan, and Huajin Tang. Enhancing snn-based spatio-temporal learning: A benchmark dataset and cross-modality attention model.Neural Networks, 180:106677, 2024

    2024

  19. [26]

    Zhenzhong Xu, Xu Chen, and Jiangtao Xu. Multi-modal multi-sensor feature fusion spiking neural network algorithm for early bearing weak fault diagnosis.Engineering Applications of Artificial Intelligence, 141:109845, 2025

    2025

  20. [27]

    Mk-sgn: A spiking graph convolutional net- work with multimodal fusion and knowledge distillation for skeleton-based action recognition.arXiv preprint arXiv:2404.10210, 2024

    Naichuan Zheng, Hailun Xia, Zeyu Liang, and Yuchen Du. Mk-sgn: A spiking graph convolutional net- work with multimodal fusion and knowledge distillation for skeleton-based action recognition.arXiv preprint arXiv:2404.10210, 2024

    2024

  21. [28]

    Snn-driven multimodal human action recognition via event camera and skeleton data fusion.arXiv preprint arXiv:2502.13385, 2025

    Naichuan Zheng and Hailun Xia. Snn-driven multimodal human action recognition via event camera and skeleton data fusion.arXiv preprint arXiv:2502.13385, 2025

    2025

  22. [29]

    Event-enhanced multi- modal spiking neural network for dynamic obstacle avoidance

    Yang Wang, Bo Dong, Yuji Zhang, Yunduo Zhou, Haiyang Mei, Ziqi Wei, and Xin Yang. Event-enhanced multi- modal spiking neural network for dynamic obstacle avoidance. InProceedings of the 31st ACM International Conference on Multimedia, pages 3138–3148, 2023

    2023

  23. [30]

    A heterogeneously integrated spiking neuron array for multimode-fused perception and object classification.Advanced Materials, 34(24):2200481, 2022

    Jiaxue Zhu, Xumeng Zhang, Rui Wang, Ming Wang, Pei Chen, Lingli Cheng, Zuheng Wu, Yongzhou Wang, Qi Liu, and Ming Liu. A heterogeneously integrated spiking neuron array for multimode-fused perception and object classification.Advanced Materials, 34(24):2200481, 2022

    2022

  24. [31]

    Sstformer: Bridging spiking neural network and memory support transformer for frame-event based recognition.arXiv preprint arXiv:2308.04369, 2023

    Xiao Wang, Zongzhen Wu, Yao Rong, Lin Zhu, Bo Jiang, Jin Tang, and Yonghong Tian. Sstformer: Bridging spiking neural network and memory support transformer for frame-event based recognition.arXiv preprint arXiv:2308.04369, 2023

    2023 arXiv

  25. [32]

    Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion.Neural Networks, 178:106493, 2024

    Hongze Sun, Rui Liu, Wuque Cai, Jun Wang, Yue Wang, Huajin Tang, Yan Cui, Dezhong Yao, and Daqing Guo. Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion.Neural Networks, 178:106493, 2024

    2024

  26. [33]

    Snn-ann hybrid networks for embedded multimodal monocular depth estimation

    Sadia Anjum Tumpa, Anusha Devulapally, Matthew Brehove, Espoir Kyubwa, and Vijaykrishnan Narayanan. Snn-ann hybrid networks for embedded multimodal monocular depth estimation. In2024 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pages 198–203. IEEE, 2024

    2024

  27. [34]

    A hybrid lstm-snn approach for robust multimodal zero-shot learning.International Journal of Complexity in Applied Science and Technology, 1(2):195–209, 2024

    Yuejia Li, Zhe Yang, Haonan Zheng, and Xiang Zhang. A hybrid lstm-snn approach for robust multimodal zero-shot learning.International Journal of Complexity in Applied Science and Technology, 1(2):195–209, 2024

    2024

  28. [35]

    What makes multi-modal learning better than single (provably).Advances in Neural Information Processing Systems, 34:10944–10956, 2021

    Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably).Advances in Neural Information Processing Systems, 34:10944–10956, 2021. 12 APREPRINT- SEPTEMBER18, 2025

    2021

  29. [36]

    Explainable multi-task learning for multi-modality biological data analysis

    Xin Tang, Jiawei Zhang, Yichun He, Xinhe Zhang, Zuwan Lin, Sebastian Partarrieu, Emma Bou Hanna, Zhaolin Ren, Hao Shen, Yuhong Yang, et al. Explainable multi-task learning for multi-modality biological data analysis. Nature communications, 14(1):2546, 2023

    2023

  30. [37]

    Learning cross-modality representations from multi-modal images.IEEE transactions on medical imaging, 38(2):638–648, 2018

    Gijs van Tulder and Marleen de Bruijne. Learning cross-modality representations from multi-modal images.IEEE transactions on medical imaging, 38(2):638–648, 2018

    2018

  31. [38]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  32. [39]

    Time-mmd: Multi-domain multimodal dataset for time series analysis.Advances in Neural Information Processing Systems, 37:77888–77933, 2024

    Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Prabhakar Kamarthi, Aditya Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, et al. Time-mmd: Multi-domain multimodal dataset for time series analysis.Advances in Neural Information Processing Syst...

    2024

  33. [40]

    Ridge estimation-based vision and laser ranging fusion localization method for uavs.Applied Optics, 64(6):1352–1361, 2025

    Huayu Huang, Chen Chen, Banglei Guan, Ze Tan, Yang Shang, Zhang Li, and Qifeng Yu. Ridge estimation-based vision and laser ranging fusion localization method for uavs.Applied Optics, 64(6):1352–1361, 2025

    2025

  34. [41]

    3d trajectory reconstruction of moving points based on a monocular camera.arXiv preprint arXiv:2502.19689, 2025

    Huayu Huang, Banglei Guan, Yang Shang, and Qifeng Yu. 3d trajectory reconstruction of moving points based on a monocular camera.arXiv preprint arXiv:2502.19689, 2025

    2025 arXiv

  35. [42]

    Full-dof egomotion estimation for event cameras using geometric solvers.arXiv preprint arXiv:2503.03307, 2025

    Ji Zhao, Banglei Guan, Zibin Liu, and Laurent Kneip. Full-dof egomotion estimation for event cameras using geometric solvers.arXiv preprint arXiv:2503.03307, 2025

    2025 arXiv

  36. [43]

    Accurate pose estimation for flight platforms based on divergent multi-aperture imaging system.arXiv preprint arXiv:2502.19708, 2025

    Shunkun Liang, Bin Li, Banglei Guan, Yang Shang, Xianwei Zhu, and Qifeng Yu. Accurate pose estimation for flight platforms based on divergent multi-aperture imaging system.arXiv preprint arXiv:2502.19708, 2025

    2025 arXiv

  37. [44]

    High-precision visual navigation device calibration method based on collimator.arXiv preprint arXiv:2502.18012, 2025

    Shunkun Liang, Dongcai Tan, Banglei Guan, Zhang Li, Guangcheng Dai, Nianpeng Pan, Liang Shen, Yang Shang, and Qifeng Yu. High-precision visual navigation device calibration method based on collimator.arXiv preprint arXiv:2502.18012, 2025

    2025 arXiv

  38. [45]

    A new 2-correspondences monocular vision navigation method under planar motion

    Yingjian Yu, Xiangyi Sun, and Banglei Guan. A new 2-correspondences monocular vision navigation method under planar motion. InJournal of Physics: Conference Series, volume 1971, page 012043. IOP Publishing, 2021

    1971

  39. [46]

    What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695–12705, 2020

    Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695–12705, 2020

    2020

  40. [47]

    Learning to balance the learning rates between various modalities via adaptive tracking factor.IEEE Signal Processing Letters, 28:1650–1654, 2021

    Ya Sun, Sijie Mai, and Haifeng Hu. Learning to balance the learning rates between various modalities via adaptive tracking factor.IEEE Signal Processing Letters, 28:1650–1654, 2021. Publisher: IEEE

    2021

  41. [48]

    Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks

    Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof J Geras. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. InInternational Conference on Machine Learning, pages 24043–24055. PMLR, 2022

    2022

  42. [49]

    Enhancing audio-visual spiking neural networks through semantic-alignment and cross-modal residual learning.arXiv preprint arXiv:2502.12488, 2025

    Xiang He, Dongcheng Zhao, Yiting Dong, Guobin Shen, Xin Yang, and Yi Zeng. Enhancing audio-visual spiking neural networks through semantic-alignment and cross-modal residual learning.arXiv preprint arXiv:2502.12488, 2025

    2025 arXiv

  43. [51]

    Temporal efficient training of spiking neural network via gradient re-weighting

    Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. InInternational Conference on Learning Representations, 2021

    2021

  44. [52]

    Esl-snns: An evolutionary structure learning strategy for spiking neural networks

    Jiangrong Shen, Qi Xu, Jian K Liu, Yueming Wang, Gang Pan, and Huajin Tang. Esl-snns: An evolutionary structure learning strategy for spiking neural networks. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 86–93, 2023

    2023

  45. [53]

    Differential coding for training-free ann-to-snn conversion.arXiv preprint arXiv:2503.00301, 2025

    Zihan Huang, Wei Fang, Tong Bu, Peng Xue, Zecheng Hao, Wenxuan Liu, Yuanhong Tang, Zhaofei Yu, and Tiejun Huang. Differential coding for training-free ann-to-snn conversion.arXiv preprint arXiv:2503.00301, 2025

    2025 arXiv

  46. [54]

    Pmr: Prototypical modal rebalance for multimodal learning

    Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20029–20038, 2023

    2023

  47. [55]

    Boosting multi-modal model performance with adaptive gradient modulation

    Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. Boosting multi-modal model performance with adaptive gradient modulation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22214–22224, 2023. 13 APREPRINT- SEPTEMBER18, 2025

    2023

  48. [56]

    Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing, 5(4):377–390, 2014

    2014

  49. [57]

    Audio-visual event localization in un- constrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in un- constrained videos. InProceedings of the European conference on computer vision (ECCV), pages 247–263, 2018

    2018

  50. [58]

    Line-based 6-dof object pose estimation and tracking with an event camera.IEEE Transactions on Image Processing, 2024

    Zibin Liu, Banglei Guan, Yang Shang, Qifeng Yu, and Laurent Kneip. Line-based 6-dof object pose estimation and tracking with an event camera.IEEE Transactions on Image Processing, 2024

    2024

  51. [59]

    Optical flow-guided 6dof object pose tracking with an event camera

    Zibin Liu, Banglei Guan, Yang Shang, Shunkun Liang, Zhenbao Yu, and Qifeng Yu. Optical flow-guided 6dof object pose tracking with an event camera. InProceedings of the 32nd ACM International Conference on Multimedia, pages 6501–6509, 2024

    2024

  52. [60]

    Stereo event-based, 6-dof pose tracking for uncooperative spacecraft.IEEE Transactions on Geoscience and Remote Sensing, 2025

    Zibin Liu, Banglei Guan, Yang Shang, Yifei Bian, Pengju Sun, and Qifeng Yu. Stereo event-based, 6-dof pose tracking for uncooperative spacecraft.IEEE Transactions on Geoscience and Remote Sensing, 2025

    2025

  53. [61]

    Lecalib: Line-based event camera calibration.Measurement, 235:114900, 2024

    Zibin Liu, Banglei Guan, Yang Shang, Zhenbao Yu, Yifei Bian, and Qifeng Yu. Lecalib: Line-based event camera calibration.Measurement, 235:114900, 2024. 14

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.