REVIEW 6 major objections 5 minor 39 references
Confidence-driven Gradient Modulation for Multimodal Human Activity Recognition: A Dynamic Contrastive Dual-Path Learning Approach
T0 review · 6 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A dual-path network with confidence-scaled gradients and multi-stage contrastive alignment aims to stop dominant sensors from silencing weaker ones in multimodal human activity recognition, reporting accuracy gains on four public sensor…
desk verdict A plausible combination of existing ideas, but the experimental protocol is fundamentally compromised by tuning on the evaluation set and by inconsistent reporting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is confidence-driven gradient modulation (CGM): after each batch, the network sums the per-sample Softmax probabilities each branch assigns to the true class, computes a dominance ratio between the branches, and for the branch with ratio above 1 multiplies its classification gradient by $1-\tanh(\alpha\,\mathrm{ReLU}(R-1))$, so the more dominant branch is damped. The alignment side is carried by a multi-stage bidirectional contrastive loss over cosine-similarity matrices at $L$ intermediate projection layers, plus an MSE loss between the $\ell^2$-normalized final branch features. The branch structure is a partitioned dual-path feature extractor: input channels are split into two disjoint sets $\mathcal{I}_1 \cup \mathcal{I}_2 = \{1,\dots,F\}$, processed by a residual-connection path and a dense-concatenation path, and globally pooled into fixed-size vectors for classification. The total objective combines three classification losses with the alignment losses as $\mathcal{L}_{total} = (\mathcal{L}_{cls,res}+\mathcal{L}_{cls,dense}+\mathcal{L}_{cls,fusion}) + \lambda_{align}(\mathcal{L}_{contrast}+\mathcal{L}_{align})$. In the authors' account, this machinery lets the two branches learn complementary features while dynamically equilibrating their contributions to the final classifier.
What would settle it
Permute the channel partition randomly over several seeds while keeping everything else fixed and compare accuracy; if a random split matches the authors' chosen split, the dual-path mechanism is not doing modality-specific work. Also track per-branch gradient norms during early training: if the confidence-driven modulation never reduces the dominant branch's gradient relative to an unmodulated run, the claimed balancing effect is not operative.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that DCDP-HAR improves multimodal human activity recognition by coupling three mechanisms: (1) a dual-path backbone in which a ResNet-style branch and a DenseNet-style branch each process a disjoint subset of the input channels, producing complementary representations; (2) a multi-stage contrastive learning scheme that treats the two branches' projected intermediate features from the same sample as positive pairs and other samples' features as negatives, averaging bidirectional contrastive losses over $L$ stages and adding an $\ell^2$-normalized MSE alignment on the final features; and (3) a confidence-driven gradient modulation that accumulates each branch's true-class Softmax confidence over a batch, forms ratios $R_{\mathrm{res}} = S_{\mathrm{res}}/(S_{\mathrm{dense}}+\epsilon)$ and $R_{\mathrm{dense}} = S_{\mathrm{dense}}/(S_{\mathrm{res}}+\epsilon)$, and scales the dominant branch's classification gradient by $1 - \tanh(\alpha\,\mathrm{ReLU}(R-1))$, with momentum-based gradient accumulation for stability. The authors report that the complete system reaches 88.34% accuracy on OPPORTUNITY, 98.75% on WISDM, 97.79% on UCI-HAR, and 93.71% on PAMAP2, beating the compared CNN, LSTM, and hybrid baselines as well as the dual-attention comparison model, and that removing the gradient modulation from the full system lowers PAMAP2 accuracy from 93.90% to 90.57%.
Load-bearing premise
The framework's central bet is that splitting the sensor channels into two fixed sets, with one set processed by a ResNet branch and the other by a DenseNet branch, produces complementary features; because the paper never says which channels go in which set, the benefit could come from simply having a bigger model.
Editorial extensions
If this is right
- If the reported results hold, modality imbalance in HAR can be treated at the gradient level rather than only through loss weighting or per-modality learning-rate schedules.
- Enforcing contrastive alignment at several intermediate stages should reduce the information loss that comes from fusing only the final feature vectors.
- The reported ablation shows the full framework at 93.90% and the same framework without gradient modulation at 90.57% on PAMAP2, so the balancing mechanism carries a substantial share of the claimed gain.
- The comparison numbers place DCDP-HAR ahead of the CNN, LSTM, and hybrid baselines on all four datasets, including the sensor-rich and class-imbalanced OPPORTUNITY set.
Reading between the lines
- The paper leaves it open which sensor channels feed the ResNet branch versus the DenseNet branch for any dataset; testing random channel partitions against the authors' implicit choice would show whether the dual-path gain is modality-specific or just extra capacity.
- The confidence-ratio rule should act most strongly early in training, so tracking per-branch gradient norms across epochs could reveal when the modulation actually changes optimization; the paper does not report that trajectory.
- The same ratio-based gradient modulation could be transferred to audio-visual or RGB-depth classification, where one modality also tends to dominate; a transfer test would be a natural extension.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DCDP-HAR, a multimodal activity-recognition framework combining a dual-path ResNet/DenseNet feature extractor, multi-stage contrastive learning for cross-modal alignment, and a confidence-driven gradient modulation (CGM) mechanism that scales branch gradients during backpropagation. The authors report ablations and comparisons on OPPORTUNITY, PAMAP2, WISDM, and UCI-HAR, claiming consistent improvements over baselines. The core claim is that the three components jointly mitigate modality imbalance and improve cross-modal alignment, leading to higher accuracy.
Significance. If the empirical claims were clean, the paper would be a modest but potentially useful contribution to multimodal sensor-based HAR: the CGM idea is a reasonable extension of gradient-modulation approaches, and multi-stage contrastive alignment is a sensible design choice. The paper also addresses a real problem, modality imbalance, and evaluates on four widely used benchmarks. However, the evaluation as presented is not trustworthy: hyperparameters are tuned on the OPPORTUNITY test set, the ablation table contains an undescribed component (DA), the sensor partition for the dual paths is never specified, and several reported numbers are inconsistent across tables. No code or detailed training protocol is provided. These issues are load-bearing for the central empirical claim, so the manuscript needs major additional work before the results can be accepted.
major comments (6)
- [Sections 5.1 and 5.2, Table 3] The headline OPPORTUNITY accuracy is not an independent test result. Section 5.1 states that hyperparameter tuning was conducted on the OPPO (OPPORTUNITY) dataset, and Section 5.2 says the finalized configuration was then used for all ablations and comparative evaluations. Because the tuning process used the test accuracy to select optimizers, architectures, contrastive weight/temperature, and the CGM alpha, the 88.34% reported for DCDP-HAR on OPPORTUNITY in Table 3 is optimistically biased by model selection on the test split. The authors must use a separate validation split or nested cross-validation for tuning and report the final test accuracy only after the configuration is fixed.
- [Tables 2 and 3, Section 5.2] Reported accuracy for the same model is inconsistent. Table 2 lists the full DCDP-HAR at 93.90% on PAMAP2, while Table 3 lists 93.71% for the same dataset and model. Similarly, Section 5.2 reports the best tuned OPPORTUNITY accuracy as 88.62%, but Table 3 reports 88.34% for DCDP-HAR on OPPORTUNITY. The authors need to clarify which configuration produced each number and which test protocol was used; as written, the discrepancies make it impossible to verify the empirical results.
- [Table 2, Section 3] The ablation study includes a component labeled 'DA' (Data Augmentation), including the row '+ DP + DA', but data augmentation is never described in the methodology. Since the full DCDP-HAR model in Table 2 includes DA, the reported 93.90% cannot be attributed solely to the three described components. The authors must specify the augmentation procedure, include it in the method, and ablate it separately with the same naming convention as the other components.
- [Section 3.1] The dual-path partition is not specified for any dataset. The paper defines feature index sets I1 and I2 for splitting the input into two complementary subsets but never states, for any benchmark, which sensor channels or feature dimensions go to the ResNet path versus the DenseNet path. Without this information, the claimed benefit of the dual-path design cannot be distinguished from simply increasing model capacity. This is especially problematic for WISDM, which is a single-sensor (accelerometer-only) dataset; the paper's framing in terms of modality imbalance and cross-modal alignment needs to explain how the dual-path split applies there.
- [Section 5.3, Table 2] The ablation discussion is internally inconsistent with the table. Adding contrastive learning to the dual-path baseline decreases accuracy from 91.71% to 91.24%, yet the text claims that removing any single module causes about a 3% accuracy reduction; in the full model, removing contrastive learning (DCDP-HAR vs +DPFE+CGM+DA) reduces accuracy by only 1.04%. The non-monotonic interaction and the mismatch between the text and the table need to be explained, and the contribution of contrastive learning should be justified in light of the negative result in the pairwise ablation.
- [Section 4, general experimental protocol] The paper does not report essential experimental details needed to reproduce or assess the results, including train/validation/test split definitions, subject-independence, window size, overlap, normalization, batch size, number of epochs, early stopping criteria, and the number of independent runs. Table 3 reports only single accuracy values without standard deviations or significance tests. The authors should provide a complete protocol and, ideally, release code to make the comparisons verifiable.
minor comments (5)
- [Sections 5.1, 5.2, 5.4] The dataset name is repeatedly written as 'OPPO' instead of 'OPPORTUNITY'; please correct this throughout.
- [References, Table 3] The text refers to 'The ELK method [39]', but reference [39] is the DanHAR paper; ELK is not introduced in the related work and has no correct citation. Please fix the reference and describe ELK properly.
- [Table 2] The row '+ DP + DA' should presumably read '+ DPFE + DA' for consistency with the other rows; please correct the abbreviation.
- [Section 5.3] The text says the ablations were performed on both PAMAP2 and OPPORTUNITY, but Table 2 does not identify the dataset and appears to show only one set of results; please clarify which dataset Table 2 refers to and provide the OPPORTUNITY ablation results if they exist.
- [Figure 1 and Section 3.3.1] The caption 'Figure 1: The total process of task' is ungrammatical and uninformative, and the section heading 'T otal Training Objective' has a typo; both should be corrected.
Circularity Check
No significant circularity: the paper's empirical components are not derived from, nor defined in terms of, the quantities they are claimed to produce.
full rationale
The paper is an empirical engineering contribution: it combines a dual ResNet/DenseNet backbone, a multi-stage contrastive alignment loss, and a confidence-scaled gradient modulation. No equation in Sections 3.1-3.4 is derived from the accuracies later reported, and no reported benchmark number is defined by construction from a fitted parameter. The hyperparameters (optimizer, architecture, contrastive weight/temperature, CGM alpha) are tuned on OPPORTUNITY in Sections 5.1-5.2 and then reused for all datasets, so the OPPORTUNITY accuracy in Table 3 is a model-selection result rather than an independent test; this is a correctness and generalization-estimate risk, not a circular derivation. The only potentially self-referential citation (ref. [14], a gradient-modulation paper by Li et al.) is used as background in the introduction and related work, not as load-bearing evidence for the present method's claims. There is no self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in by citation. The ablation study is a heuristic component analysis, not a prediction that reduces to its inputs. The Section 5.4 'ELK method[39]' citation mismatch is a reference error, not a circularity issue. Hence the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- alpha =
0.9
- tau =
0.5
- lambda_align =
0.7
assumptions (3)
- domain assumption The input sensor channels can be partitioned into two mutually exclusive subsets such that processing them independently with ResNet and DenseNet yields complementary features.
- ad hoc to paper The softmax confidence for the true class is a valid measure of a modality's contribution to learning, and suppressing the more confident branch improves balance.
- domain assumption The momentum-based gradient accumulation (Equation with beta) stabilizes training under gradient modulation; the specific beta value is not justified.
Cite this review
Pith. "Pith review of Confidence-driven Gradient Modulation for Multimodal Human Activity Recognition: A Dynamic Contrastive Dual-Path Learning Approach." pith.science (2026). https://pith.science/paper/YIIGPZY2
@misc{pith2026250702826,
author = {Pith},
title = {Pith review of: Confidence-driven Gradient Modulation for Multimodal Human Activity Recognition: A Dynamic Contrastive Dual-Path Learning Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/YIIGPZY2}},
note = {Machine review of arXiv:2507.02826}
}
read the original abstract
Sensor-based Human Activity Recognition (HAR) is a core technology that enables intelligent systems to perceive and interact with their environment. However, multimodal HAR systems still encounter key challenges, such as difficulties in cross-modal feature alignment and imbalanced modality contributions. To address these issues, we propose a novel framework called the Dynamic Contrastive Dual-Path Network (DCDP-HAR). The framework comprises three key components. First, a dual-path feature extraction architecture is employed, where ResNet and DenseNet branches collaboratively process multimodal sensor data. Second, a multi-stage contrastive learning mechanism is introduced to achieve progressive alignment from local perception to semantic abstraction. Third, we present a confidence-driven gradient modulation strategy that dynamically monitors and adjusts the learning intensity of each modality branch during backpropagation, effectively alleviating modality competition. In addition, a momentum-based gradient accumulation strategy is adopted to enhance training stability. We conduct ablation studies to validate the effectiveness of each component and perform extensive comparative experiments on four public benchmark datasets.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[39]
W. Gao, L. Zhang, Q. Teng, J. He, H. Wu, Danhar: Dual attention network for multimodal human activity recognition using wearable sensors, Applied Soft Computing 111 (2021) 107728. First Author et al.: Preprint submitted to Elsevier Page 13 of 13
work page 2021
- [1]
-
[2]
X. Chen, Y . Xiao, Y . Tang, J. Fernandez-Mendoza, G. Cao, Apneadetector: Detecting sleep apnea with smartwatches, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5 (2) (2021) 1–22
work page 2021
- [3]
-
[4]
K. Bayoudh, R. Knani, F. Hamdaoui, A. Mtibaa, A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets, The Visual Computer 38 (8) (2022) 2939–2970
work page 2022
-
[5]
S. K. Yadav, K. Tiwari, H. M. Pandey, S. A. Akbar, A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions, Knowledge-Based Systems 223 (2021) 106970
work page 2021
-
[6]
Y . Yin, L. Xie, Z. Jiang, F. Xiao, J. Cao, S. Lu, A systematic review of human activity recognition based on mobile devices: overview, progress and trends, IEEE Communications Surveys & Tutorials 26 (2) (2024) 890–929
work page 2024
-
[7]
L. Bao, S. S. Intille, Activity recognition from user-annotated acceleration data, in: International conference on pervasive computing, Springer, 2004, pp. 1–17
work page 2004
Show all 39 references
-
[8]
N. Ravi, N. Dandekar, P. Mysore, M. L. Littman, Activity recognition from accelerometer data, in: Aaai, V ol. 5, Pittsburgh, PA, 2005, pp. 1541–1546. First Author et al.: Preprint submitted to Elsevier Page 12 of 13 Short Title of the Article
2005
-
[9]
Zhang, Y
S. Zhang, Y . Li, S. Zhang, F. Shahabi, S. Xia, Y . Deng, N. Alshurafa, Deep learning in human activity recognition with wearable sensors: A review on advances, Sensors 22 (4) (2022) 1476
2022
-
[10]
M. Z. Uddin, A. Soylu, Human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learning, Scientific Reports 11 (1) (2021) 16455
2021
-
[11]
J. Wang, Y . Chen, S. Hao, X. Peng, L. Hu, Deep learning for sensor-based activity recognition: A survey, Pattern recognition letters 119 (2019) 3–11
2019
-
[12]
Ehatisham-Ul-Haq, A
M. Ehatisham-Ul-Haq, A. Javed, M. A. Azam, H. M. Malik, A. Irtaza, I. H. Lee, M. T. Mahmood, Robust human activity recognition using multimodal feature-level fusion, IEEE Access 7 (2019) 60736–60751
2019
-
[13]
A. Das, S. Das, G. Sistu, J. Horgan, U. Bhattacharya, E. Jones, M. Glavin, C. Eising, Revisiting modality imbalance in multimodal pedestrian detection, in: 2023 IEEE International Conference on Image Processing (ICIP), IEEE, 2023, pp. 1755–1759
2023
-
[14]
H. Li, X. Li, P. Hu, Y . Lei, C. Li, Y . Zhou, Boosting multi-modal model performance with adaptive gradient modulation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22214–22224
2023
-
[15]
Dufumier, J
B. Dufumier, J. Castillo-Navarro, D. Tuia, J.-P. Thiran, What to align in multimodal contrastive learning?, arXiv preprint arXiv:2409.07402 (2024)
2024 arXiv
-
[16]
W. Wang, D. Tran, M. Feiszli, What makes training multi-modal classification networks hard?, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 12695–12705
2020
-
[17]
Fujimori, R
N. Fujimori, R. Endo, Y . Kawai, T. Mochizuki, Modality-specific learning rate control for multimodal classification, in: Pattern Recognition: 5th Asian Conference, ACPR 2019, Auckland, New Zealand, November 26–29, 2019, Revised Selected Papers, Part II, V ol. 5, Springer Inte...
2019
-
[18]
Y . Fan, W. Xu, H. Wang, J. Wang, S. Guo, Pmr: Prototypical modal rebalance for multimodal learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 20029–20038
2023
-
[19]
Zhang, J
X. Zhang, J. Yoon, M. Bansal, H. Yao, Multimodal representation learning by alternating unimodal adaptation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 27456–27466
2024
-
[20]
H. Ma, W. Li, X. Zhang, S. Gao, S. Lu, Attnsense: Multi-level attention mechanism for multimodal human activity recognition., in: IJCAI, 2019, pp. 3109–3115
2019
-
[21]
X. Peng, Y . Wei, A. Deng, D. Wang, D. Hu, Balanced multimodal learning via on-the-fly gradient modulation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 8238–8247
2022
-
[22]
Chung, J
S. Chung, J. Lim, K. J. Noh, G. Kim, H. Jeong, Sensor data acquisition and multimodal sensor fusion for human activity recognition using deep learning, Sensors 19 (7) (2019) 1716
2019
-
[23]
Alharbi, L
F. Alharbi, L. Ouarbya, J. A. Ward, Comparing sampling strategies for tackling imbalanced data in human activity recognition, Sensors 22 (4) (2022) 1373
2022
-
[24]
A. K. Koupai, M. J. Bocus, R. Santos-Rodriguez, R. J. Piechocki, R. McConville, Self-supervised multimodal fusion transformer for passive activity recognition, IET Wireless Sensor Systems 12 (5-6) (2022) 149–160
2022
-
[25]
Hotelling, Relations between two sets of variates, in: Breakthroughs in statistics: methodology and distribution, Springer, 1992, pp
H. Hotelling, Relations between two sets of variates, in: Breakthroughs in statistics: methodology and distribution, Springer, 1992, pp. 162–190
1992
-
[26]
Andrew, R
G. Andrew, R. Arora, J. Bilmes, K. Livescu, Deep canonical correlation analysis, in: International Conference on Machine Learning, PMLR, 2013, pp. 1247–1255
2013
-
[27]
W. Wang, R. Arora, K. Livescu, J. Bilmes, On deep multi-view representation learning, in: International Conference on Machine Learning, PMLR, 2015, pp. 1083–1092
2015
-
[28]
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, Y . Bengio, Show, attend and tell: Neural image caption generation with visual attention, in: International Conference on Machine Learning, PMLR, 2015, pp. 2048–2057
2015
-
[29]
R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, Y . Bengio, Learning deep representations by mutual information estimation and maximization, arXiv preprint arXiv:1808.06670 (2018)
2018 arXiv
-
[30]
T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, in: International Conference on Machine Learning, PMLR, 2020, pp. 1597–1607
2020
-
[31]
Radford, J
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning, PMLR, 2021, pp. 8748–8763
2021
-
[32]
Chavarriaga, H
R. Chavarriaga, H. Sagha, A. Calatroni, S. T. Digumarti, G. Tröster, J. d. R. Millán, D. Roggen, The opportunity challenge: A benchmark database for on-body sensor-based activity recognition, Pattern Recognition Letters 34 (15) (2013) 2033–2042
2013
-
[33]
Reiss, D
A. Reiss, D. Stricker, Introducing a new benchmarked dataset for activity monitoring, in: 2012 16th international symposium on wearable computers, IEEE, 2012, pp. 108–109
2012
-
[34]
J. R. Kwapisz, G. M. Weiss, S. A. Moore, Activity recognition using cell phone accelerometers, ACM SigKDD Explorations Newsletter 12 (2) (2011) 74–82
2011
-
[35]
Anguita, A
D. Anguita, A. Ghio, L. Oneto, X. Parra, J. L. Reyes-Ortiz, et al., A public domain dataset for human activity recognition using smartphones., in: Esann, V ol. 3, 2013, pp. 3–4
2013
-
[36]
M. Zeng, L. T. Nguyen, B. Yu, O. J. Mengshoel, J. Zhu, P. Wu, J. Zhang, Convolutional neural networks for human activity recognition using mobile sensors, in: 6th international conference on mobile computing, applications and services, IEEE, 2014, pp. 197–205
2014
-
[37]
K. Xia, J. Huang, H. Wang, Lstm-cnn architecture for human activity recognition, Ieee Access 8 (2020) 56855–56866
2020
-
[38]
N. Dua, S. N. Singh, V . B. Semwal, Multi-input cnn-gru based human activity recognition using wearable sensors, Computing 103 (7) (2021) 1461–1478
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.