Pith. sign in

REVIEW 3 major objections 5 minor 43 references

VSMP-IMU: Video-Grounded Semantic Motion Programs for Sensor-Aware Synthetic IMU Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VSMP-IMU argues that synthetic IMU data for wearable activity recognition is best produced through a structured Semantic Motion Program that separates activity-defining invariants from label-preserving variants, and reports 78.33 average…

desk verdict Honest, carefully evaluated system paper; the SMP representation claim survives its own matched ablation, but the +4.04 headline over prior baselines is partly confounded by a class-faithfulness filter that only VSMP-IMU receives. read the letter →

arxiv 2608.05782 v1 pith:SBUBLNTM submitted 2026-08-06 cs.CV cs.LG

Lala Shakti Swarup Ray , Vitor Fortes Rey , Mengxi Liu , Paul Lukowicz , Bo Zhou This is my paper · ORCID
classification cs.CVcs.LG
keywords syntheticIMUgenerationSemanticMotionProgramhumanactivityrecognitionvideo-to-IMUsensorgroundingtext-to-motionsynthesiswearablesensorslong-taillearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to establish that synthetic inertial (IMU) data for wearable human activity recognition is most useful when generated through a structured intermediate representation, not by transferring noisy pose estimates from video or by prompting from coarse text. The proposed framework, VSMP-IMU, extracts from an activity video a Semantic Motion Program (SMP) that separates what defines the activity's identity from what can safely vary, augments that program into label-preserving variants, verbalizes each variant for a text-to-motion model, converts the generated 3D motion into virtual IMU signals, and grounds those signals to the statistics of the target wearable dataset. If the central claim is right, the field gains a practical answer to labeled-data scarcity: 78.33 average Macro-F1 across five datasets under leave-one-person-out evaluation, 9.77 points above real-only training, 4.04 above the strongest prior synthetic baseline, and larger margins at 1% label budgets and for tail classes. The paper also documents that its own control knobs weaken between generation and sensor: requested tempo and repetition attenuate after grounding, while amplitude survives comparatively better.

What carries the argument

The load-bearing object is the Semantic Motion Program (SMP), a structured schema describing an activity through motion primitives, body parts, temporal organization, execution attributes, object/contact cues, and per-field uncertainty, with an explicit partition into invariant fields (activity label, primitive order, mandatory body parts, required contact) and variant fields (tempo, amplitude, repetition count, primitive duration, dominant side, symmetry, displacement). It carries the argument in two ways. First, it makes augmentation semantic rather than signal-level: variant fields are perturbed under an uncertainty-scaled radius $\Delta_f(u_f)=(1-u_f)\Delta_f^{\max}$ with clipping, and categorical changes are gated by both uncertainty and activity-specific rules, so the sampled programs stay within the class label. Second, it is the interface between video understanding and synthesis: the video-grounded SMP is verbalized into text-to-motion prompts, and because the frozen video model and the frozen motion generator never talk directly, the SMP is what keeps the synthesized motion anchored to the extracted activity structure.

What would settle it

Run the full VSMP-IMU pipeline with a deliberately corrupted invariant/variant split — for example, marking a known label-defining primitive as a variant field and perturbing it exactly as the augmentation rules would — and check the 78.33 average Macro-F1: if the class-faithfulness filter (trained only on real data) removes the drifted samples and the average holds, the filter, not the SMP split, is doing the load-bearing work; if performance collapses, the split itself carries the result. A companion check is the correlation between per-class SMP extraction accuracy (74% primitive-sequence accuracy on the hand-centric subset) and per-class F1 gain: near-zero correlation would imply extraction quality is not the binding factor.

Watch

Extended reading notes

Core claim

VSMP-IMU claims that a video-to-IMU pipeline built on an explicit Semantic Motion Program produces synthetic wearable data that is more useful for HAR than real-only training, classical signal augmentation, direct video-to-IMU transfer, or text-only synthesis. The SMP records the activity label, ordered motion primitives, primitive durations, body parts, temporal structure, execution attributes, object/contact cues, augmentation constraints, and per-field uncertainty, and splits these into invariants (which define the class and are never changed) and variants (which may be perturbed under an uncertainty-scaled radius). From an input video, a frozen video-capable vision-language model extracts the SMP, rule-based consistency checks repair it, constrained augmentation samples $K$ variants, each is verbalized into a prompt for an off-the-shelf text-to-motion model, the motion is converted to virtual IMU, and the signals are adapted to the target wearable domain by simulation-in-the-loop optimization followed by rank-based per-channel distribution matching, then filtered for motion plausibility, class faithfulness, and redundancy. The reported evidence: 78.33 average Macro-F1 over real-only's 68.56 and IMUTube's 74.30 under standard leave-one-person-out evaluation across MM-Fit, UTD-MHAD, MMAct, PAMAP2, and HAD-AW; low-resource gains of 18.54 over real-only; long-tail tail-class gains of 19.86 over real-only; and a controllability study on 12 UTD-MHAD activities in which requested tempo correlates at $\rho=0.92$ with generated motion but only $\rho=0.61$ with grounded IMU, repetition counts land within one repetition in 90.6% of motions, and primitive-duration error rises from 9.8% in motion to 14.1% after grounding.

Load-bearing premise

The load-bearing premise is that the frozen vision-language model's split of SMP fields into 'safe to vary' versus 'defines the activity' is correct: the paper's own audit reports variant-field Macro-F1 of only 0.84, and 15.5% of audited clips retain at least one incorrect field after refinement, so if a label-defining field is mislabeled as augmentable, perturbing it silently drifts the synthetic sample toward a different activity.

Editorial extensions

If this is right

  • If the framework is right, labeled IMU scarcity stops being a hard wall: at a 1% label budget VSMP-IMU reports +19.95 Macro-F1 over real-only training, and gains remain positive up to the 50% budget.
  • Class imbalance is partly fixable at the data level: tail-class Macro-F1 improves by 19.86 over real-only and 4.76 over the strongest prior synthetic baseline, while head classes do not lose performance.
  • Structure, not just richer prompts, is what helps: under strictly matched attribute phrases, the SMP beats a generic caption by 3.26 Macro-F1, with main effects of +2.90 (structure) and +1.90 (variation) and a positive interaction.
  • Semantic control is real but bounded by sensor observability: tempo correlates at $\rho=0.92$ with generated motion but only $\rho=0.61$ with grounded IMU, so downstream value depends on placing the virtual sensor where the varied attribute leaves a signal.
  • The benefit generalizes across activity types — wrist-dominant and non-periodic actions gain +4.4 and +4.5 over the strongest baseline — but stops short of articulated finger motion and contact-transient classes such as catch and knock.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension the paper leaves implicit: the SMP is a modality-neutral interface, so the same invariant/variant programs could drive synthetic data for other sensor modalities (heart-rate chest straps, pressure floors, earbud inertial sensors), not just IMU.
  • The measured grounding attenuation (e.g., tempo $\rho$ from 0.92 to 0.61) suggests the real bottleneck for synthetic-data value is sensor observability rather than semantic extraction; a testable consequence is that choosing source videos or virtual sensor placements per attribute would recover part of the lost correlation.
  • Because offline cost is about 10 seconds per sample, the economically sensible unit is a cached per-activity SMP pool reused across downstream models and target datasets; the paper does not develop this amortization argument.
  • The 15.5% residual extraction-error rate invites a sharper decomposition than the paper's ablations provide: quantify how much of the final gain is produced by the class-faithfulness filter rejecting drifted samples versus by the structured representation itself, for example by removing the filter at matched retained budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents VSMP-IMU, a video-grounded framework for synthetic IMU generation built around a structured Semantic Motion Program (SMP). The SMP is extracted from video with a frozen VLM, refined by rule checks, augmented under uncertainty-conditioned label-preserving constraints, verbalized for the HY-Motion text-to-motion model, converted to virtual IMU signals with IMUSim, and finally grounded to the target wearable domain via simulation-in-the-loop optimization, target-format alignment, rank-based marginal matching, and a three-component quality filter. The evaluation covers standard, low-resource, and long-tail leave-one-person-out protocols on MM-Fit, UTD-MHAD, MMAct, PAMAP2, and HAD-AW, with four downstream backbones. The headline result is an average Macro-F1 of 78.33, a gain of 9.77 over real-only training and 4.04 over the strongest prior synthetic baseline, together with a controllability study on 12 UTD-MHAD activities and an SMP extraction audit on 200 manually annotated clips.

Significance. If the headline comparison is clean, the contribution is significant: a structured intermediate representation that separates activity-defining invariants from label-preserving variation, combined with sensor-aware grounding, would be a practical advance for synthetic IMU generation in low-resource and long-tail HAR. The paper has several genuine strengths that I want to acknowledge explicitly: the LOPO protocols are careful about leakage (all grounding statistics, validation thresholds, and filtering components are restricted to non-test subjects); the strictly matched caption–SMP ablation (Section 4.5.1, Table 14) controls for attribute requests, source clips, seeds, retained sample counts, and downstream settings; the SMP extraction audit (Section 4.4) is transparent about residual errors; and the controllability study (Section 4.3) uses matched prompts and paired seeds to separate representation-level control from motion-level and sensor-level realization. The reproducibility appendix is unusually detailed.

major comments (3)
  1. [Section 3.8, Eq. (34); Section 4.1.4; Table 5] The class-faithfulness filter is applied to VSMP-IMU's synthetic samples, using a HAR validator trained on the real training fold to discard any sample whose top-1 prediction is not the intended label or whose confidence is below tau_cls. Section 4.1.4 states that all generation-based methods share the same downstream backbone, training schedule, synthetic sample budget, and source video pool, but it never states that IMUTube and IMUGPT 2.0 are subjected to the same motion-plausibility, class-faithfulness, and redundancy filters. If the baselines contribute all generated samples while VSMP-IMU can discard drifted or hard samples, then the +4.04 headline gain over the strongest prior synthetic baseline is not attributable to the SMP representation alone; it may partly reflect the selection advantage of the filter. I agree with the stress-test concern on this point. The internal caption–SMP ablation controls for filtering by design, but that does not resolve the baseline comparison. Please either apply the same filtering procedure to the baselines, or report the effect of the filter by comparing VSMP-IMU with and without filtering against each baseline with and without the same filtering. This is required for the central claim in the abstract.
  2. [Section 4.2.1, Table 5] The paper reports that VSMP-IMU achieves the best Macro-F1 in 19 of 20 dataset–backbone combinations and an average improvement of 4.04 over the strongest baseline, but no significance tests are reported. Some of the per-entry differences are within one standard deviation of the competing method (for example, PAMAP2 with RF: IMUTube 86.46±2.38 vs. VSMP-IMU 87.92±1.84; PAMAP2 with DCLSTM: 82.48±3.18 vs. 83.91±1.43). Since results are averaged over five seeds and all LOPO folds, paired statistical tests across folds or seeds, or confidence intervals, are needed to support the 'best in 19 of 20' and the average-gain claims. Without this, the strength of the headline result is overstated.
  3. [Section 4.2.3, Section 4.1.3] The long-tail protocol is ambiguous: Section 4.1.3 says 'Synthetic data is generated either for tail classes only or in proportion to class scarcity,' but Section 4.2.3 does not state which choice was used for the reported results. The tail-class gain of 19.86 over real-only training is a central part of the paper's contribution, and the outcome depends materially on whether synthetic data is generated only for tail classes or for all classes. Please specify the exact protocol used in Table 7 and, ideally, report both variants. The same ambiguity affects the comparison with IMUTube and IMUGPT 2.0 in the long-tail setting.
minor comments (5)
  1. [Section 4.6, caption of Figure 4] The words 'as visulaized in Figure 4' contain a typo; it should read 'as visualized in Figure 4.'
  2. [Section 3.8, Eq. (34)] The paper does not report the actual tau_cls values selected per fold or per dataset, despite the appendix stating that tau_cls is chosen from {0.50, 0.55, 0.60, 0.65, 0.70} using validation Macro-F1. Reporting the selected values and the fraction of samples rejected by each filter component would make the quality-filter behavior more transparent.
  3. [Section 4.1.2] The source-video collection procedure mentions YouTube candidates and manual verification, but it does not report the number of candidates retrieved, the number discarded at each filtering stage, or the final number of clips per class. Since the source-video pool is shared across video-based methods, a brief accounting would help reproducibility.
  4. [Section 4.5.2, Table 15] The ablation table reports that rank-based grounding alone gives 83.1 on PAMAP2 while simulation-in-the-loop only gives 69.8. Given that the rank transform is a per-channel marginal alignment, it is surprising that it produces such a large gain on an activity with strong temporal dynamics; a short explanation of why marginal alignment alone is so effective on PAMAP2 would be helpful.
  5. [Section 5.3] The limitations paragraph repeats the sentence about tempo and repetition count that also appears in Section 4.3.6 and in the Conclusion; the duplicate passage should be removed or condensed.

Circularity Check

1 steps flagged · score 2.0 of 10

No fundamental circularity: held-out test-subject evaluation and external baselines anchor the result. One mild self-referential step: the class-faithfulness filter (Eq.

  1. other [Section 3.8, Eq. 34 (quality filtering; class-faithfulness) and Section 4.1.4 (baseline sharing statement)]
    "Given a grounded synthetic sequence bXimu, the validator outputs class probabilities p(c| bXimu). Synthetic samples with confidence below a threshold τcls or with a predicted label different from y are discarded: Qclass(bXimu,y)=I[max_c p(c| bXimu) ≥ τcls ∧ arg max_c p(c| bXimu)=y]."

    The retained synthetic set is defined, by Eq. 34, as the subset that a lightweight HAR validator trained only on the same real training folds already classifies as the intended label with confidence above τcls. The headline gain (+4.04 over the strongest prior synthetic baseline) is therefore partly produced by this selection step rather than by the video-grounded SMP semantics alone, and Section 4.1.4 states only that generation-based methods share backbone, schedule, budget, and source video pool — not that IMUTube and IMUGPT 2.0 receive the identical three-component quality filter. This is a mild self-referential selection step, not a fit of test labels: no held-out-subject data enters the filter, the threshold, or the grounding statistics.

full rationale

Walking the derivation chain — video → VLM-extracted SMP → constrained augmentation → verbalization → HY-Motion motion synthesis → IMUSim simulation → simulation-in-the-loop + rank-based grounding → quality filtering → downstream HAR training → held-out-subject Macro-F1 — the central result does not reduce to its inputs. Downstream Macro-F1 is measured on held-out LOPO test subjects; all grounding statistics (D_stat, D_rank) and the filter validator are estimated only from training folds (Sections 3.7, 3.8, 4.1.3); no test label enters any fitted parameter. HY-Motion and IMUSim are external or frozen backends, and the two strongest baselines (IMUTube, IMUGPT 2.0) are independent prior methods, so the comparison is anchored outside the present paper. The single quotable self-referential element is the class-faithfulness filter (Eq. 34): retained synthetic samples are, by construction, those a same-fold-trained validator already labels as the intended class, and Section 4.1.4 does not state that the prior synthetic baselines receive the identical three-component filter; this partially shapes the +4.04 headline gain and warrants the flagged caveat. However, the strictly matched caption–SMP ablation (Section 4.5.1) applies identical filters, seeds, retained counts, and verbatim attribute phrases across C0/C1/S0/S1 and still finds a 3.26-average gain for the constrained SMP over the matched caption, so the structured-representation claim is independently supported. The paper's own limitations (Section 5.3) honestly acknowledge the 15.5% residual refined-SMP error and its potential propagation, and the end-to-end controllability study measures requested-vs-realized agreement directly rather than assuming it. Self-citations (Rey/Lukowicz prior video-to-IMU work) are background only and are not load-bearing for the central claim. Overall the derivation is substantially self-contained against external benchmarks; the identified issue is a selection-and-fairness caveat on the cross-method headline, not an equivalence of the claimed result to its inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The performance claims are empirical, so the main ledger items are hand-set augmentation ranges, calibration weights, filter thresholds, and mixing ratios. The SMP itself is the only introduced conceptual object; it is partially validated by the audit and controllability study.

free parameters (5)
  • SMP augmentation perturbation ranges (tempo, amplitude, repetition, primitive duration, side, symmetry, displacement) = tempo 1 level, amplitude 1 level, repetitions 5 cycles, duration 25%, side p=0.5, symmetry p=0.3, displacement 25%
    Hand-chosen in Table 18; they control how much each variant field may move during augmentation and therefore shape synthetic diversity.
  • Simulation-in-the-loop objective weights and search ranges = lambda_Sigma=1.0, lambda_psd=0.5, lambda_r=0.5; orientation +/-10-15 deg, bias +/-0.5 m/s^2 and +/-0.05 rad/s, noise…
    Chosen for CEM optimization in Section A.4; they determine how strongly synthetic IMU is pulled toward training-fold temporal and multivariate statistics.
  • Class-faithfulness filter threshold tau_cls = 0.60 selected from {0.50, 0.55, 0.60, 0.65, 0.70} by validation Macro-F1
    Section A.6; controls how many synthetic samples are discarded, and together with the redundancy threshold defines the training set composition.
  • Synthetic-to-real mixing ratio alpha = 0.5 for MM-Fit, UTD-MHAD, MMAct; 0.25 for PAMAP2 and HAD-AW
    Selected per dataset on validation (Table 16); high ratios reduce performance, so the reported gains depend on this tuning.
  • VLM frame sampling and query count = 16-32 frames, 512x512 resolution, five queries per clip
    Implementation choices in Appendix A.1 and A.2 that affect extraction quality and cost.
assumptions (4)
  • domain assumption GPT-5.3 video VLM can reliably return schema-compliant SMP JSON from monocular clips and its five responses can be aggregated without systematic bias.
    Used in Eq. (2) and Section 3.3; the audit shows 96.5% label accuracy after refinement but residual variant-field errors remain.
  • domain assumption HY-Motion text-to-motion synthesis realizes verbalized SMP variants with sufficient kinematic fidelity for virtual IMU synthesis, including wrist trajectories but not finger articulation.
    Section 3.6 and controllability results show partial realization (tempo rho=0.92 motion-level, 0.61 after grounding), so this is a bounded assumption.
  • domain assumption IMUSim's kinematic-to-inertial conversion and the subsequent rank-based grounding are adequate models of real wearable sensors.
    Sections 3.7 and 3.8; gains on PAMAP2 support this, but no independent sensor-placement validation is provided.
  • domain assumption Training-fold statistics and the class-faithfulness validator can be used to filter and ground synthetic data without leaking held-out subject information.
    Protocol in Sections 3.8 and 4.1.3; the design is leakage-controlled, but the filter creates a selection bias relative to baselines.
invented entities (1)
  • Semantic Motion Program (SMP) intermediate representation independent evidence
    purpose: Encodes activity label, primitives, body parts, temporal structure, execution attributes, contact, constraints, and uncertainty to control synthetic IMU generation.
    Not a physical entity but a novel conceptual construct; the paper supplies extraction-audit and controllability evidence, though invariant/variant assignment remains imperfect (variant-field Macro-F1 0.84).

how reviews work

0 comments
Cite this review

Pith. "Pith review of VSMP-IMU: Video-Grounded Semantic Motion Programs for Sensor-Aware Synthetic IMU Generation." pith.science (2026). https://pith.science/paper/SBUBLNTM

@misc{pith2026260805782,
  author       = {Pith},
  title        = {Pith review of: VSMP-IMU: Video-Grounded Semantic Motion Programs for Sensor-Aware Synthetic IMU Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBUBLNTM}},
  note         = {Machine review of arXiv:2608.05782}
}
read the original abstract

Wearable human activity recognition (HAR) is often limited by the scarcity of labeled sensor data, especially in low-resource, class-imbalanced, and subject-generalization settings. Synthetic IMU generation can reduce this dependency and enhance HAR machine learning model's performance, but existing approaches face a trade-off without addressing all factors: video-driven methods are visually grounded but sensitive to pose-estimation errors, while text-driven methods are controllable but often weakly grounded in how activities are actually performed. We present VSMP-IMU, a video-grounded framework for controllable synthetic IMU generation based on a structured Semantic Motion Program (SMP), which separates activity-defining semantics from label-preserving variation. Given an input video, VSMP-IMU extracts and augments an SMP, uses it to synthesize motion, converts the motion into virtual IMU signals, and grounds the resulting signals to the target wearable domain. We evaluate VSMP-IMU against state-of-the-art synthetic data generation methods on five public IMU-HAR datasets under leave-one-person-out evaluation. VSMP-IMU achieves an average Macro-F1 of 78.33%, improving over real-only training by 9.77% and over the strongest prior synthetic baseline by 4.04%. In low-resource settings with reduced training data-samples, it improves over real-only training by 18.54% and over the strongest prior synthetic baselines by more than 6% on average. Under long-tail evaluation in imbalanced datasets, it improves tail-class Macro-F1 by 19.86% over Real-only training and by 4.76% over SOTA. These results show that structured video-grounded semantics provide a practical foundation for controllable, wearable-relevant synthetic sensor data generation.

Figures

Figures reproduced from arXiv: 2608.05782 by the authors.

Figure 1
Figure 1. Compared to alternative synthetic IMU data generation methods, VSMP-IMU better preserves activity-defining [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of VSMP-IMU. VSMP-IMU extracts a video-grounded SMP from an activity video, refines and augments it under activity-preserving constraints, verbalizes each variant for HY-Motion synthesis, converts the generated motion into virtual IMU signals, and grounds and filters the signals to produce the final synthetic sensor dataset. simulator parameter 𝜂 contains only bounded physical corrections to this nominal si… view at source ↗
Figure 3
Figure 3. End-to-end SMP controllability on UTD-MHAD. (a) Requested tempo versus measured motion tempo and grounded-IMU tempo. (b) Requested versus detected repetition count in generated motion and grounded IMU. The dashed line denotes perfect count agreement. (c) Requested amplitude versus generated joint range of motion and grounded-IMU energy. (d) Confusion matrix for motion-level symmetric and asymmetric requests. Points … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Feature-space realism analysis on the UTD-MHAD dataset. Real UTD-MHAD samples and synthetic samples generated [PITH_FULL_IMAGE:figures/full_fig_p035_4.png]
Figure 5
Figure 5. Figure 5: Detailed case study for a single activity. Starting from a source video clip, VSMP-IMU extracts an SMP and generates three controlled variants by modifying selected execution attributes while preserving activity identity and core primitive structure. The figure shows t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 26 canonical work pages

  1. [1]

    Sara Ashry, Reda Elbasiony, and Walid Gomaa. 2018. An LSTM-based descriptor for human activities recognition using IMU sensors. In Proceedings of the 15th International Conference on Informatics in Control, Automation and Robotics, ICINCO, Vol. 1. 494–501

    work page 2018

  2. [2]

    Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz. 2015. UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In2015 IEEE International conference on image processing (ICIP). IEEE, 168–172

    work page 2015

  3. [3]

    Eadom Dessalene, Michael Maynord, Cornelia Fermüller, and Yiannis Aloimonos. 2023. Leap: Llm-generation of egocentric action programs.arXiv preprint arXiv:2312.00055(2023)

    work page Pith review arXiv 2023

  4. [4]

    Eadom Dessalene, Michael Maynord, Cornelia Fermüller, and Yiannis Aloimonos. 2023. Therbligs in action: Video understanding through motion primitives. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10618–10626. , Vol. 1, No. 1, Article . Publication date: August 2026. 38•Kim et al

    work page 2023

  5. [5]

    Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu. 2023. Motion question answering via modular motion programs. InInternational Conference on Machine Learning. PMLR, 9312–9328

    work page 2023

  6. [6]

    Vitor Fortes Rey, Kamalveer Kaur Garewal, and Paul Lukowicz. 2021. Translating videos into synthetic training data for wearable sensor-based activity recognition systems using residual deep convolutional networks.Applied Sciences11, 7 (2021), 3094

    2021

  7. [7]

    Lars Ole Haeusler, Lena Uhlenberg, and Oliver Amft. 2025. Text2IMU: Advancing Human Activity Recognition by Text-Driven IMU Data Synthesis. In2025 IEEE 21st International Conference on Body Sensor Networks (BSN). IEEE, 1–4

    work page 2025

  8. [8]

    Harish Haresamudram, Apoorva Beedu, Mashfiqui Rabbi, Sankalita Saha, Irfan Essa, and Thomas Ploetz. 2025. Limitations in employing natural language supervision for sensor-based human activity recognition-and ways to overcome them. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 273–281

    work page 2025

Show all 43 references
  1. [9]

    Harish Haresamudram, Irfan Essa, and Thomas Plötz. 2021. Contrastive predictive coding for human activity recognition.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies5, 2 (2021), 1–26

    2021

  2. [10]

    Harish Haresamudram, Irfan Essa, and Thomas Plötz. 2022. Assessing the state of self-supervised human activity recognition using wearables.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies6, 3 (2022), 1–47

    2022

  3. [11]

    Harish Haresamudram, Chi Ian Tang, Sungho Suh, Paul Lukowicz, and Thomas Ploetz. 2025. Past, present, and future of sensor-based human activity recognition using wearables: A surveying tutorial on a still challenging task.Proceedings of the ACM on Interactive, Mobile, Wearable...

    2025

  4. [12]

    Zhiqing Hong, Zelong Li, Shuxin Zhong, Wenjun Lyu, Haotian Wang, Yi Ding, Tian He, and Desheng Zhang. 2024. Crosshar: Generalizing cross-dataset human activity recognition via hierarchical self-supervised pretraining.Proceedings of the ACM on Interactive, Mobile, Wearable and ...

    2024

  5. [13]

    Yash Jain, Hyeokhyen Kwon, and Thomas Ploetz. 2022. On the effectiveness of virtual imu data for eating detection with wrist sensors. In Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symp...

    2022

  6. [14]

    Chi Yoon Jeong, Hyung Cheol Shin, and Mooseop Kim. 2021. Sensor-data augmentation for human activity recognition with time-warping and data masking.Multimedia Tools and Applications80, 14 (2021), 20991–21009

    2021

  7. [15]

    Hua Kang, Qianyi Huang, and Qian Zhang. 2022. Augmented adversarial learning for human activity recognition with partial sensor sets.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies6, 3 (2022), 1–30

    2022

  8. [16]

    Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami. 2019. Mmact: A large-scale dataset for cross modal human action understanding. InProceedings of the IEEE/CVF International Conference on Computer Vision. 8658–8667

    2019

  9. [17]

    Sumith Kulal, Jiayuan Mao, Alex Aiken, and Jiajun Wu. 2021. Hierarchical motion understanding via motion programs. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6568–6576

    2021

  10. [18]

    Hyeokhyen Kwon, Gregory D Abowd, and Thomas Plötz. 2021. Complex deep neural networks from large scale virtual imu data for effective human activity recognition using wearables.Sensors21, 24 (2021), 8337

    2021

  11. [19]

    Hyeokhyen Kwon, Catherine Tong, Harish Haresamudram, Yan Gao, Gregory D Abowd, Nicholas D Lane, and Thomas Ploetz. 2020. Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition.Proceedings of the ACM on Interactive, Mobile, Wear...

    2020

  12. [20]

    Hyeokhyen Kwon, Bingyao Wang, Gregory D Abowd, and Thomas Plötz. 2021. Approaching the real-world: Supporting activity recognition training with virtual imu data.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies5, 3 (2021), 1–32

    2021

  13. [21]

    Arttu Lämsä, Jaakko Tervonen, Jussi Liikka, Constantino Álvarez Casado, and Miguel Bordallo López. 2022. Video2IMU: Realistic IMU features and signals from videos. In2022 IEEE-EMBS International Conference on Wearable and Implantable Body Sensor Networks (BSN). IEEE, 1–5

    2022

  14. [22]

    Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, and Thomas Plötz. 2024. Imugpt 2.0: Language-based cross modality transfer for sensor-based human activity recognition.Proceedings of the ACM on Interactive, Mobile, Wearabl...

    2024

  15. [23]

    Zikang Leng, Archith Iyer, and Thomas Plötz. 2025. Scaling Human Activity Recognition: A Comparative Evaluation of Synthetic Data Generation and Augmentation Techniques.arXiv preprint arXiv:2506.07612(2025)

    2025 arXiv

  16. [24]

    Zikang Leng, Yash Jain, Hyeokhyen Kwon, and Thomas Ploetz. 2023. On the utility of virtual on-body acceleration data for fine-grained human activity recognition. InProceedings of the 2023 ACM International Symposium on Wearable Computers. 55–59

    2023

  17. [25]

    Zikang Leng, Hyeokhyen Kwon, and Thomas Plötz. 2023. Generating virtual on-body accelerometer data from virtual textual descriptions for human activity recognition. InProceedings of the 2023 ACM International Symposium on Wearable Computers. 39–43

    2023

  18. [26]

    Jiyang Li, Lin Huang, Siddharth Shah, Sean J Jones, Yincheng Jin, Dingran Wang, Adam Russell, Seokmin Choi, Yang Gao, Junsong Yuan, et al. 2023. Signring: Continuous american sign language recognition using imu rings and virtual imu data.Proceedings of the ACM on Interactive, ...

    2023

  19. [27]

    Wang Lu, Jindong Wang, Yiqiang Chen, Sinno Jialin Pan, Chunyu Hu, and Xin Qin. 2022. Semantic-discriminative mixup for generalizable sensor-based cross-domain activity recognition.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies6, 2 , Vol. 1,...

    2022

  20. [28]

    Shenghuan Miao and Ling Chen. 2024. Goat: A generalized cross-dataset activity recognition framework with natural language supervision.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies8, 4 (2024), 1–28

    2024

  21. [29]

    Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition.Sensors16, 1 (2016), 115

    2016

  22. [30]

    Xin Qin, Yiqiang Chen, Jindong Wang, and Chaohui Yu. 2019. Cross-dataset activity recognition via adaptive spatial-temporal transfer learning.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies3, 4 (2019), 1–25

    2019

  23. [31]

    Attila Reiss and Didier Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In2012 16th international symposium on wearable computers. IEEE, 108–109

    2012

  24. [32]

    Vitor Fortes Rey, Peter Hevesi, Onorina Kovalenko, and Paul Lukowicz. 2019. Let there be IMU data: generating training data for wearable, motion sensor based activity recognition from monocular RGB videos. InAdjunct proceedings of the 2019 ACM international joint conference on...

    2019

  25. [33]

    Panneer Selvam Santhalingam, Parth Pathak, Huzefa Rangwala, and Jana Kosecka. 2023. Synthetic smartwatch imu data generation from in-the-wild asl videos.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies7, 2 (2023), 1–34

    2023

  26. [34]

    Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. 2020. Finegym: A hierarchical video dataset for fine-grained action understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2616–2625

    2020

  27. [35]

    Satya P Singh, Madan Kumar Sharma, Aimé Lay-Ekuakille, Deepak Gangwar, and Sukrit Gupta. 2020. Deep ConvLSTM with self-attention for human activity decoding using wearable sensors.IEEE Sensors Journal21, 6 (2020), 8575–8582

    2020

  28. [36]

    David Strömbäck, Sangxia Huang, and Valentin Radu. 2020. Mm-fit: Multimodal deep learning for automatic exercise logging across sensing devices.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies4, 4 (2020), 1–22

    2020

  29. [37]

    Jie Su, Zhenyu Wen, Tao Lin, and Yu Guan. 2022. Learning disentangled behaviour patterns for wearable-based human activity recognition.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies6, 1 (2022), 1–19

    2022

  30. [38]

    Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. 2017. Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks. InProceedings of the 19th ACM internat...

    2017

  31. [39]

    Tianshi Wang, Yizhuo Chen, Qikai Yang, Dachun Sun, Ruijie Wang, Jinyang Li, Tomoyoshi Kimura, and Tarek Abdelzaher. 2024. Data augmentation for human activity recognition via condition space interpolation within a generative model. In2024 33rd International Conference on Compu...

    2024

  32. [40]

    Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. 2024. Move as you say interact as you can: Language-guided human motion generation with scene affordance. InProceedings of the IEEE/CVF Conference on Co...

    2024

  33. [41]

    Yuxin Wen, Qing Shuai, Di Kang, Jing Li, Cheng Wen, Yue Qian, Ningxin Jiao, Changhai Chen, Weijie Chen, Yiran Wang, et al. 2025. HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation.arXiv preprint arXiv:2512.23464(2025)

    2025

  34. [42]

    Junjie Zhang, Yuanhao Liu, and Hua Yuan. 2023. Attention-based residual BiLSTM networks for human activity recognition.IEEE Access11 (2023), 94173–94187

    2023

  35. [43]

    activity_label

    Yexu Zhou, Haibin Zhao, Yiran Huang, Tobias Röddiger, Murat Kurnaz, Till Riedel, and Michael Beigl. 2024. AutoAugHAR: automated data augmentation for sensor-based human activity recognition.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies8, 2...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.