Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

SensorLM: Learning the Language of Wearable Sensors

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SensorLM claims that automatically generated hierarchical captions from 59.7 million hours of wearable data are enough to align sensor streams with natural language, achieving state-of-the-art zero-shot activity recognition, few-shot…

desk verdict Big new sensor-text pretraining resource and a sensible captioning pipeline, but the zero-shot/retrieval claims are confounded by deterministic captions and unclear test-set separation. read the letter →

arxiv 2506.09108 v1 pith:KUPSGVJO submitted 2025-06-10 cs.LG cs.AIcs.CL

Yuwei Zhang , Kumar Ayush , Siyuan Qiao , A. Ali Heydari , Girish Narayanswamy , Maxwell A. Xu , Ahmed A. Metwally , Shawn Xu
show 12 more authors
Jake Garrison Xuhai Xu Tim Althoff Yun Liu Pushmeet Kohli Jiening Zhan Mark Malhotra Shwetak Patel Cecilia Mascolo Xin Liu Daniel McDuff Yuzhe Yang
This is my paper
classification cs.LGcs.AIcs.CL
keywords wearablesensorssensor-languagefoundationmodelhierarchicalcaptiongenerationzero-shotactivityrecognitioncross-modalretrievalfew-shotlearninghealthpredictioncontrastive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Wearable sensors record a day of heart rate, motion, skin temperature, and electrodermal activity, but turning those numbers into words is hard because paired sensor-text data is scarce. This paper tries to remove that bottleneck by manufacturing captions at scale: a hierarchical pipeline converts each day of sensor data into statistical summaries, structural pattern descriptions, and semantic event captions, producing 59.7 million hours of paired data from over 103,000 people. On top of that corpus the paper trains SensorLM, a family of models that combine a contrastive alignment objective with an autoregressive captioning objective. The reported result is that this recipe yields state-of-the-art zero-shot activity recognition, strong few-shot transfer to health prediction, near-perfect cross-modal retrieval, and the ability to caption sensor data and generalize to unseen activities. A sympathetic reader would care because it proposes a practical, scalable route from raw physiological streams to natural-language health and activity understanding.

What carries the argument

The load-bearing mechanism is the hierarchical caption-generation pipeline. It verbalizes each day of 26 minutely aggregated sensor features into three caption levels: statistical captions reporting means, extremes, and standard deviations; structural captions describing trends, spikes, and fluctuations with time windows; and semantic captions naming activity episodes, sleep periods, and logged mood with start and end minutes. These captions are paired with the sensor windows and used to train a sensor encoder, a text encoder, and a multimodal text decoder under a weighted combination of a symmetric contrastive loss and an autoregressive captioning loss. The same template style is reused at evaluation time for prompt ensembling and retrieval queries, which helps the model align sensor embeddings with language embeddings.

What would settle it

Build a held-out set whose captions are written by humans, or generated by an independent pipeline, from the same sensor days, and re-evaluate retrieval and zero-shot recognition on it; if recall@1 and AUROC collapse on human-written text, the model has learned the template distribution rather than sensor semantics. A second check: shuffle event timestamps in the captions and see whether retrieval still identifies the correct sensor days; if it does, the model is ignoring temporal structure.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single architecture trained with both a contrastive loss and a captioning loss on hierarchical template captions can give wearable sensor encoders genuine language understanding. On 20-class activity recognition, SensorLM reaches 0.84 AUROC zero-shot while LLM baselines fed the same data as tables stay near random; with only 50 labeled examples per class it reaches 0.88 AUROC. Cross-modal retrieval is near-perfect, with recall at rank 1 between 96 and 100 percent on query sets up to 40,000 items in both sensor-to-text and text-to-sensor directions. The authors also report that the model generates coherent captions for unseen days, groups conceptually similar activities in embedding space, and generalizes to activities absent from pretraining, such as snowboarding, because of proximity to known concepts. In the paper's framing, the generic pretraining objective recovers earlier contrastive and hybrid vision-language pretraining schemes as specific configurations, making SensorLM a generalization of that paradigm to sensors.

Load-bearing premise

The load-bearing premise is that template-generated captions extracted from the same sensor signals are a sufficient language supervision signal, so that near-perfect retrieval and strong zero-shot scores reflect sensor-language understanding rather than a learned mapping from features to caption strings.

Editorial extensions

If this is right

  • Zero-shot activity understanding: the model labels 20 activities and abstract concepts such as outdoor versus indoor sports without any task-specific training, where LLM baselines with tabularized sensor data perform near random.
  • Label-efficient health transfer: frozen sensor embeddings support linear probes and few-shot heads, reaching 0.88 AUROC on 20-class activity with 50 labels per class and improving hypertension and anxiety prediction over self-supervised baselines.
  • Reliable cross-modal retrieval: at 100 candidates retrieval is perfect and at 40,000 candidates recall@1 stays above 90 percent in both directions, enabling natural-language queries over sensor archives.
  • Generative and zero-shot capabilities: the model produces captions for unseen sensor days and recognizes unseen activities such as snowboarding by proximity to related training concepts.
  • Predictable scaling: zero-shot performance improves with compute, data, and model size, with diminishing returns beyond roughly 12 million hours of data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial caution: because the caption pipeline and the retrieval benchmark use the same template generator, the near-perfect retrieval scores may partly measure how well the model reproduces a deterministic feature-to-text mapping; a benchmark with human-written captions would test true language generalization.
  • The recipe is domain-portable: any time-series domain with expert vocabularies, such as ECG episodes, EEG events, or motion primitives, could get a sensor-language model by writing hierarchical templates without needing human annotation at scale.
  • If the scaling-law trend holds, the next bottleneck is caption diversity rather than raw data volume, so generating richer semantic captions, for example rarer activities or more granular event boundaries, may yield larger gains than adding more hours of the same signals.
  • A stricter test of the model's semantics would be to scramble event timestamps in the retrieval captions; if retrieval still succeeds, the model is matching event names rather than temporal structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SensorLM, a family of sensor-language foundation models trained on a large corpus of wearable sensor data with automatically generated hierarchical captions. The pretraining dataset comprises 59.7 million hours from more than 103,000 people, and the model architecture combines contrastive and generative objectives, recovering CLIP, Cap, and CoCa as special cases. The authors report strong zero-shot activity recognition, few-shot transfer to activity and health tasks, near-perfect cross-modal retrieval, and sensor caption generation, along with scaling and generalization analyses.

Significance. If the claims are upheld, the scale of the dataset and the unified architecture would be a useful step for wearable sensor foundation models, and the few-shot and linear probing results on the held-out Metabolic dataset provide some independent evidence of useful representations. However, the headline zero-shot and retrieval results are weakened by evaluation-design issues: the captions are generated deterministically from the same sensor data, and the Activity test set is not explicitly stated to be held out from pretraining. The paper would be significant after a careful re-evaluation on strictly held-out data and with evaluation material that is not produced by the same caption-generation pipeline.

major comments (3)
  1. [Section 5.1, Table 7] The Activity evaluation does not state that the test data were held out from pretraining, and the reported counts are internally inconsistent. Section 5.1 describes the Activity dataset as 22,289 person-days from 10,013 individuals and notes only that it is 'from the same population as our pretraining sensor data,' while Table 7 lists Activity train/test participant counts of 37,822 and 8,343, which sum to 46,165 and do not match the 10,013 figure. Since the pretraining cohort (103,643 participants; 2,489,570 person-days) is far larger and no exclusion is explicit, the zero-shot results in Table 2 and the few-shot results in Fig. 5 could reflect memorization of pretraining person-days rather than generalization. Please specify exactly how Activity participants and person-days were disjoint from pretraining, reconcile Table 7, and re-report the main results on a verified held-out split. In addition, clarify whether the activity detector that generates semantic captions was trained on data that includes the Activity test participants.
  2. [Section 5.1, Section 3.2, Table 21] The retrieval benchmark is circular with respect to the pretraining signal. The retrieval captions are generated by the same hierarchical pipeline described in Section 3.2, and the captions are deterministic functions of the input sensor statistics, detected trends, and activity events. Near-perfect R@1 values (100% at 100 samples; 96.1% sensor-to-text at 40k in Table 21) may therefore indicate that the model has learned to invert the template mapping, not that it has achieved robust language alignment. Please add retrieval experiments with captions written by humans or paraphrased through a different generation process, report the number of distinct caption templates in the retrieval set, and show retrieval performance as a function of caption novelty. Without such evidence, the cross-modal retrieval claim is not an independent demonstration of sensor-language understanding.
  3. [Section 5.1, Table 2, Appendix B.8] The main zero-shot classification evaluation uses the same activity vocabulary that appears in the semantic captions used during pretraining, so it should be framed as classification within the pretraining label space rather than as evidence of generalization to unseen concepts. The 'zero-shot generalization to unseen classes' experiment in Appendix B.8 is a separate case study with a retrained model and is the appropriate test of generalization; the main model's 20-class results should not be described as zero-shot generalization without qualification. I also recommend reporting whether the semantic caption generator's activity labels are obtained from a supervised model that may have seen the test participants, since that would further compromise the independence of the zero-shot evaluation.
minor comments (5)
  1. [Appendix C.1, Fig. 8] The axis labels in Fig. 8 are garbled with sequences such as '/uni00000231/uni00000230/uni00000232' and the figure is unreadable; it should be regenerated.
  2. [Tables 9, 11; Fig. 10; Fig. 3] There are several typos: 'An decreasing trend' appears in Tables 9 and 11, 'stasistics' appears in the Fig. 10 caption, and 'SensorLMarchitecture' appears in the Fig. 3 label.
  3. [Section 5.2, Appendix B.9] The caption generation evaluation is described as using 'a dedicated evaluation set of 200 sensor-text pairs (sampled from the sensor-caption retrieval dataset),' but the sampling procedure and the exact pretraining configuration of the 12-decoder-layer model used for captioning are not specified; please provide these details.
  4. [Fig. 5, Tables 4, 5, 19] Several reported differences are small (e.g., Anxiety AUROC 0.65 for SensorLM vs 0.66 for MSN in Table 19), yet no confidence intervals or significance tests are provided; adding them would make the few-shot and ablation comparisons more convincing.
  5. [General] The manuscript does not include a data or code availability statement; given the emphasis on the scale of the dataset and the reproducibility of the captioning pipeline, a clear statement would be helpful.

Circularity Check

2 steps flagged · score 6.0 of 10

Retrieval and Activity benchmarks reuse the same caption/activity generator that defines the pretraining targets, so the headline numbers partly reduce to fitting that generator.

  1. self definitional [Section 5, 'Datasets', 'Sensor-text retrieval dataset'; Section 3.2 'Hierarchical Sensor Caption Generation'; Section 4 pretraining objectives]
    "Sensor-text retrieval dataset. We construct a retrieval dataset of 39,766 examples sourced from a separate set of 975 individuals who were not included in the pretraining set. The same selection criteria and caption generation method used in pretraining were applied."

    The retrieval ground-truth captions are generated by the same deterministic caption pipeline (Section 3.2) that produced the pretraining targets, and those captions are verbalizations of statistics, detected trends, and recognized activities computed from the sensor input x itself. Training maximizes log p(g(x)|x) via Lcap and aligns s(x) with v(g(x)) via Lcon, where g is the caption generator. Retrieval therefore ranks exactly the training pair type (x, g(x)) on held-out subjects; near-perfect R@1 (96-100%) measures how well the model learned the generator's input-output mapping, which is the training objective by construction. The captions are not an independent sensor-language ground truth, so the retrieval claim partially reduces to self-definition.

  2. fitted input called prediction [Section 5.1, 'Datasets', 'Activity dataset'; Table 7]
    "The Activity dataset comprises 22,289 person-days from 10,013 individuals. We randomly sampled∼1,000 test examples for each activity for zero-shot activity recognition (AR) and few-shot adaptation. This dataset is from the same population as our pretraining sensor data."

    Unlike the retrieval and Metabolic sets, the paper never states that the Activity participants or person-days were excluded from the 2,489,570 person-day pretraining set; it only says 'same population'. If any test person-days were seen with their semantic captions during pretraining, the zero-shot AUROC (0.84), unseen-class generalization, and few-shot probing could reflect memorization of the caption/activity mapping rather than sensor-language alignment. Table 7 aggravates the ambiguity by reporting Activity downstream train/test totals of 37,822/8,343 participants, inconsistent with the 10,013 individuals stated in Section 5.1. The absence of an exclusion statement leaves the central zero-shot claim unsupported as a genuine holdout evaluation.

full rationale

SensorLM's central evaluation loop is partially closed by construction. The pretraining captions are generated from the sensor inputs via a fixed hierarchical pipeline (statistical summaries, detected trends, recognized activities), and the cross-modal retrieval benchmark applies the same caption generation method to build its ground truth. Since the target caption is a deterministic function of the input, high retrieval accuracy demonstrates that the model learned the caption generator's mapping, not that it has independently validated sensor-language understanding. The Activity zero-shot/few-shot evaluation is similarly vulnerable: the paper describes the dataset as coming 'from the same population as our pretraining sensor data' without stating that the test person-days were held out, and Table 7's participant counts are internally inconsistent. These are concrete, quotable reductions of the headline claims to the training setup, not mere authorial self-citation. The architectural unification (CLIP/Cap/CoCa variants) and the scaling analyses are not circular, and no load-bearing uniqueness theorem is imported from prior work. Overall, the central empirical claims are partially circular, warranting a score of 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. The main assumed ingredients are the sufficiency of the 26 aggregated features, the accuracy of activity and sleep labels, and the validity of template captions as language supervision. These are domain assumptions rather than fitted parameters, but they are load-bearing for the claimed capabilities.

free parameters (5)
  • Contrastive and captioning loss weights (lambda_con, lambda_cap) = 1, 1
    Chosen without reported search; they balance the two pretraining objectives and directly affect learned representations.
  • Temperature tau in contrastive loss = 0.01
    Set as a hyperparameter; controls the sharpness of the softmax distribution and influences embedding alignment.
  • ViT patch size for sensor encoder = (2, 10)
    Chosen by hand; determines the tokenization of the 26 x 1440 input matrix.
  • Trend and spike detection thresholds in structural caption generation = not reported
    These thresholds decide which trends and spikes are verbalized; they are not specified in the paper and affect the text that the model learns from.
  • Number of caption templates per level = 20 statistical, 15 structural, 20 semantic
    Template counts and the random selection procedure shape caption diversity, but the exact template sets are only partially shown.
assumptions (4)
  • domain assumption 26 minutely aggregated features are sufficient to represent the sensor signals; high-frequency raw data can be discarded.
    Appendix A.1 states that raw waveforms are not used due to storage and compute constraints. If the aggregated features lose critical information, the model's understanding of activities and health states would be limited.
  • domain assumption Activity, sleep, and mood labels used in semantic captions are accurate ground truth.
    Section 3.2 builds semantic captions from 'recognized activities and sleep periods' and user-logged mood. These labels come from the wearable system or user self-report and are treated as correct without validation.
  • ad hoc to paper Template-based captions provide a sufficient training signal for sensor-language alignment.
    The core premise of the paper is that automatically generated templated text can substitute for human-written descriptions. If this fails, the model only learns a fixed template-to-feature mapping.
  • standard math Transformer architectures and contrastive/captioning losses work for sensor time series as they do for images and text.
    The paper adopts established VLP methods (CLIP, CoCa) with a ViT encoder; this relies on the standard behavior of these components for a new modality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SensorLM: Learning the Language of Wearable Sensors." pith.science (2026). https://pith.science/paper/KUPSGVJO

@misc{pith2026250609108,
  author       = {Pith},
  title        = {Pith review of: SensorLM: Learning the Language of Wearable Sensors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KUPSGVJO}},
  note         = {Machine review of arXiv:2506.09108}
}
read the original abstract

We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SleepLM: Natural-Language Intelligence for Human Sleep

    cs.AI 2026-02 conditional novelty 7.0 of 10

    A sleep-language foundation model trained with contrastive, captioning, and reconstruction objectives outperforms general LLMs and fine-tuned VLMs on zero-shot sleep staging, event localization, and cross-modal retrieval.

  2. HEARTS: Benchmarking LLM Reasoning on Health Time Series

    cs.LG 2026-02 conditional novelty 7.0 of 10

    A 110-task benchmark across 20 health signal modalities shows current LLMs underperform specialized models and depend on simple heuristics rather than robust time-series reasoning.

  3. Signal or Noise? Understanding Generative Models for Real-World Sensor Time Series

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Across 14 sensor generation settings, flow-matching models are the strongest overall baseline, while demographic conditioning, time-frequency modeling, and moderate synthetic augmentation improve hard regimes and down...

  4. OSF: On Pre-training and Scaling of Sleep Foundation Models

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Channel-masked self-supervised pretraining on a 166,500-hour multi-source sleep corpus yields OSF, which generalizes better to missing channels and scales with data and model size.

Reference graph

Works this paper leans on

69 extracted references · 44 canonical work pages · cited by 4 Pith papers

  1. [1]

    Large-scale training of foundation models for wearable biosignals

    Salar Abbaspourazad, Oussama Elachqar, Andrew Miller, Saba Emrani, Udhyakumar Nallasamy, and Ian Shapiro. Large-scale training of foundation models for wearable biosignals. InThe Twelfth International Conference on Learning Representations, 2023

    work page 2023

  2. [2]

    Masked siamese networks for label-efficient learning

    MahmoudAssran, MathildeCaron, IshanMisra, PiotrBojanowski, FlorianBordes, PascalVincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. InEuropean Conference on Computer Vision, pages 456–473. Springer, 2022

    work page 2022

  3. [3]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021

    2021

  4. [4]

    Bootstrap confidence intervals: when, which, what? a practical guide for medical statisticians.Statistics in medicine, 19(9):1141–1164, 2000

    James Carpenter and John Bithell. Bootstrap confidence intervals: when, which, what? a practical guide for medical statisticians.Statistics in medicine, 19(9):1141–1164, 2000

    work page 2000

  5. [5]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on machine learning, pages 1597–1607. PMLR, 2020

    2020

  6. [6]

    Wenqiang Chen, Jiaxuan Cheng, Leyao Wang, Wei Zhao, and Wojciech Matusik. Sensor2text: Enabling natural language interactions for daily activity tracking using wearable sensors.Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(4):1–26, 2024

    work page 2024

  7. [7]

    Towards a personal health large language model.arXiv preprint arXiv:2406.06474, 2024

    Justin Cosentino, Anastasiya Belyaeva, Xin Liu, Nicholas A Furlotte, Zhun Yang, Chace Lee, Erik Schenck, Yojan Patel, Jian Cui, Logan Douglas Schneider, et al. Towards a personal health large language model.arXiv preprint arXiv:2406.06474, 2024

    arXiv 2024

  8. [8]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

Show all 69 references
  1. [9]

    A visual– language foundation model for pathology image analysis using medical twitter.Nature medicine, 29(9):2307–2316, 2023

    Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual– language foundation model for pathology image analysis using medical twitter.Nature medicine, 29(9):2307–2316, 2023

    2023

  2. [10]

    Llasa: A multimodal llm for human activity analysis through wearable and smartphone sensors.arXiv preprint arXiv:2406.14498, 2024

    Sheikh Asif Imran, Mohammad Nur Hossain Khan, Subrata Biswas, and Bashima Islam. Llasa: A multimodal llm for human activity analysis through wearable and smartphone sensors.arXiv preprint arXiv:2406.14498, 2024

    2024

  3. [11]

    Neurolm: A universal multi-task foundation model for bridging the gap between language and eeg signals

    Weibang Jiang, Yansen Wang, Bao-liang Lu, and Dongsheng Li. Neurolm: A universal multi-task foundation model for bridging the gap between language and eeg signals. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  4. [12]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  5. [13]

    Health-llm: Large languagemodelsforhealthpredictionviawearablesensordata

    Yubin Kim, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, and Hae Won Park. Health-llm: Large languagemodelsforhealthpredictionviawearablesensordata. arXivpreprintarXiv:2401.06866 , 2024. 12 SensorLM: Learning the Language of Wearable Sensors

    2024 arXiv

  6. [14]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  7. [15]

    Zechen Li, Shohreh Deldari, Linyao Chen, Hao Xue, and Flora D. Salim. SensorLLM: Aligning large language models with motion sensors for human activity recognition, 2025

    2025

  8. [16]

    Large language models are few-shot health learners.arXiv preprint arXiv:2305.15525, 2023

    Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming- Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners.arXiv preprint arXiv:2305.15525, 2023

    2023 arXiv

  9. [17]

    Guidelines for augmentation selection in contrastive learning for time series classification.arXiv preprint arXiv:2407.09336, 2024

    Ziyu Liu, Azadeh Alavi, Minyi Li, and Xiang Zhang. Guidelines for augmentation selection in contrastive learning for time series classification.arXiv preprint arXiv:2407.09336, 2024

    2024 arXiv

  10. [18]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2017

    2017

  11. [19]

    A visual-language foundation model for computational pathology.Nature Medicine, 30(3):863–874, 2024

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology.Nature Medicine, 30(3):863–874, 2024

    2024

  12. [20]

    Transforming wearable data into health insights using large language model agents.arXiv preprint arXiv:2406.06464, 2024

    Mike A Merrill, Akshay Paruchuri, Naghmeh Rezaei, Geza Kovacs, Javier Perez, Yun Liu, Erik Schenck, Nova Hammerquist, Jake Sunshine, Shyam Tailor, et al. Transforming wearable data into health insights using large language model agents.arXiv preprint arXiv:2406.06464, 2024

    2024 arXiv

  13. [21]

    Merrill, Mingtian Tan, Vinayak Gupta, Thomas Hartvigsen, and Tim Althoff

    Mike A. Merrill, Mingtian Tan, Vinayak Gupta, Thomas Hartvigsen, and Tim Althoff. Language models still struggle to zero-shot reason about time series. InEMNLP (Findings), pages 3512– 3533, 2024

    2024

  14. [22]

    Imu2clip: language-grounded motion sensor translation with multimodal con- trastive learning

    Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Damavandi. Imu2clip: language-grounded motion sensor translation with multimodal con- trastive learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13246...

    2023

  15. [23]

    Tailor, Jacob Sunshine, Yun Liu, Tim Althoff, Shrikanth Narayanan, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Samy Abdel-Ghaffar, and Daniel McDuff

    Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, shun liao, Jake Garri- son, Shyam A. Tailor, Jacob Sunshine, Yun Liu, Tim Althoff, Shrikanth Narayanan, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Samy Abdel-Ghaffar, and Daniel McDuff. Scaling w...

    2025

  16. [24]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  17. [25]

    Fitbit-based interventions for healthy lifestyle outcomes: systematic review and meta-analysis.Journal of medical Internet research, 22(10):e23954, 2020

    Mickael Ringeval, Gerit Wagner, James Denford, Guy Paré, and Spyros Kitsiou. Fitbit-based interventions for healthy lifestyle outcomes: systematic review and meta-analysis.Journal of medical Internet research, 22(10):e23954, 2020

    2020

  18. [26]

    Data augmentation for learning predictive models on eeg: a systematic comparison.Journal of Neural Engineering, 19(6):066020, 2022

    Cédric Rommel, Joseph Paillard, Thomas Moreau, and Alexandre Gramfort. Data augmentation for learning predictive models on eeg: a systematic comparison.Journal of Neural Engineering, 19(6):066020, 2022

    2022

  19. [27]

    Exploringcontrastive learning in human activity recognition for healthcare.arXiv preprint arXiv:2011.11542, 2020

    ChiIanTang, IgnacioPerez-Pozuelo, DimitrisSpathis, andCeciliaMascolo. Exploringcontrastive learning in human activity recognition for healthcare.arXiv preprint arXiv:2011.11542, 2020. 13 SensorLM: Learning the Language of Wearable Sensors

    2011 arXiv

  20. [28]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  21. [29]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  22. [30]

    SleepFM: Multi-modal representation learning for sleep across brain activity, ECG and respiratory signals

    Rahul Thapa, Bryan He, Magnus Ruud Kjaer, Hyatt Moore IV, Gauri Ganjoo, Emmanuel Mignot, and James Zou. SleepFM: Multi-modal representation learning for sleep across brain activity, ECG and respiratory signals. InForty-first International Conference on Machine Learning, 2024

    2024

  23. [31]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  24. [32]

    Image captioners are scalable vision learners too.Advances in Neural Information Processing Systems, 36:46830–46855, 2023

    Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too.Advances in Neural Information Processing Systems, 36:46830–46855, 2023

    2023

  25. [33]

    Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    2008

  26. [34]

    Simvlm: Simple visual language model pretraining with weak supervision.arXiv preprint arXiv:2108.10904, 2021

    ZiruiWang, JiahuiYu, AdamsWeiYu, ZihangDai, YuliaTsvetkov, andYuanCao. Simvlm: Simple visual language model pretraining with weak supervision.arXiv preprint arXiv:2108.10904, 2021

    2021 arXiv

  27. [35]

    Deepsqa: Understanding sensor data via question answering

    Tianwei Xing, Luis Garcia, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. Deepsqa: Understanding sensor data via question answering. InProceedings of the International Conference on Internet-of-Things Design and Implementation, pages 106–118, 2021

    2021

  28. [36]

    Simper: Simple self-supervised learning of periodic targets

    Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, and Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. In The Eleventh International Conference on Learning Representations, 2023

    2023

  29. [37]

    Artificial intelligence-enabled detection and assessment of parkinson’s disease using nocturnal breathing signals.Nature Medicine, 28(10):2207–2215, 2022

    Yuzhe Yang, Yuan Yuan, Guo Zhang, Hao Wang, Ying-Cong Chen, Yingcheng Liu, Christopher G Tarolli, Daniel Crepeau, Jan Bukartyk, Mithri R Junna, et al. Artificial intelligence-enabled detection and assessment of parkinson’s disease using nocturnal breathing signals.Nature Medic...

    2022

  30. [38]

    Coca: Contrastivecaptionersareimage-textfoundationmodels

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastivecaptionersareimage-textfoundationmodels. arXivpreprintarXiv:2205.01917 , 2022

    2022 arXiv

  31. [39]

    Sensorchat: Answering qualitative and quantitative questions during long-term multimodal sensor interactions.arXiv preprint arXiv:2502.02883, 2025

    Xiaofan Yu, Lanxiang Hu, Benjamin Reichman, Dylan Chu, Rushil Chandrupatla, Xiyuan Zhang, Larry Heck, and Tajana Rosing. Sensorchat: Answering qualitative and quantitative questions during long-term multimodal sensor interactions.arXiv preprint arXiv:2502.02883, 2025

    2025 arXiv

  32. [40]

    Self-supervised learning for human activity recognition using 700,000 person-days of wearable data.NPJ digital medicine, 7(1):91, 2024

    Hang Yuan, Shing Chan, Andrew P Creagh, Catherine Tong, Aidan Acquah, David A Clifton, and Aiden Doherty. Self-supervised learning for human activity recognition using 700,000 person-days of wearable data.NPJ digital medicine, 7(1):91, 2024

    2024

  33. [41]

    Self-supervised contrastive pre-training for time series via time-frequency consistency.Advances in neural information processing systems, 35:3988–4003, 2022

    Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency.Advances in neural information processing systems, 35:3988–4003, 2022. 14 SensorLM: Learning the Language of Wearable Sensors

    2022

  34. [42]

    Unimts: Unified pre-training for motion time series.Advances in Neural Information Processing Systems, 37:107469–107493, 2024

    Xiyuan Zhang, Diyan Teng, Ranak Roy Chowdhury, Shuheng Li, Dezhi Hong, Rajesh Gupta, and Jingbo Shang. Unimts: Unified pre-training for motion time series.Advances in Neural Information Processing Systems, 37:107469–107493, 2024

    2024

  35. [43]

    Ecg-chat: A large ecg-language model for cardiac disease diagnosis.arXiv preprint arXiv:2408.08849, 2024

    Yubao Zhao, Tian Zhang, Xu Wang, Puyu Han, Tong Chen, Linlin Huang, Youzhu Jin, and Jiaju Kang. Ecg-chat: A large ecg-language model for cardiac disease diagnosis.arXiv preprint arXiv:2408.08849, 2024

    2024 arXiv

  36. [44]

    Heart rate

    Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al. A founda- tion model for generalizable disease detection from retinal images.Nature, 622(7981):156–163, 2...

    2023

  37. [45]

    The average Heart rate value is 88.7, with extremes at 134.9 (max) and 70.8 (min), and a std of 9.3

  38. [46]

    The Heart rate data exhibits a mean of 88.7, a standard deviation of 9.3, and its extreme values are 70.8 and 134.9

  39. [47]

    Heart rate average 88.7, reaching a maximum of 134.9 and a minimum of 70.8, with a standard deviation of 9.3

  40. [48]

    Heart rate exhibits a mean of 88.7, with peak and minimal values reaching 134.9 and 70.8, and a standard deviation of 9.3

  41. [49]

    For the Heart rate measurements, the mean is 88.7, the standard deviation is 9.3, and the data lies between 70.8 and 134.9. ...... A.4. Data Acquisition and Approval The data used for training in our analysis was curated from a large corpus of historical wearable data collecte...

  42. [51]

    Heart rate exhibits decreasing trend during minute 680-960 interval

  43. [52]

    An decreasing trend in Heart rate data recorded between minute 680 and 960

  44. [53]

    The Heart rate trend from minute 680 to 960 is decreasing

  45. [54]

    From minute 680 to 960, Heart rate exhibits an decreasing trend. ...... Spikes:

  46. [55]

    Spike event recorded for steps at minute 720

  47. [56]

    Data indicates a peak for steps at the 720-minute mark

  48. [57]

    Minute 720 shows a spike for the steps

  49. [58]

    A peak is detected for steps at minute 720

  50. [59]

    The steps experienced a spike at minute 720. ...... Table 10| Example of prompt templates used in semantic captions. Semantic Caption Templates

  51. [60]

    From minute 1121 to 1133, the user had a period of Outdoor Bike

  52. [61]

    Outdoor Bike recorded within the 1121-1133 minute range

  53. [62]

    Outdoor Bike episode occurred between minute 1121 and 1133

  54. [63]

    Outdoor Bike was recorded between minute 1121 and 1133

  55. [64]

    Evaluate the feasibility of using the data provided by wrist-worn wearable devices to develop algorithms and scores to assess metabolic health

    Identified Outdoor Bike across the timeframe of minute 1121 to 1133. ...... consent language described use of the data for developing new health features and algorithms and being included in publications:“Fitbit will collect and use your data to research and develop new health...

  56. [65]

    A period of Run was observed during the session

  57. [66]

    Detected a phase of Run

  58. [67]

    Data shows Run took place

  59. [68]

    The main action was Run

  60. [69]

    Activity by environmental context

    Run was detected during the observed period. ...... B.5. Zero-Shot Classification B.5.1. Task definition for zero-shot classification We formulate six zero-shot classification tasks related to activity recognition. We compareSensorLM with representative LLM baselines acrossmai...

  61. [730]

    Anxiety” and “Hypertension

    From minute 446 to 456, the user had a period of Outdoor Bike. Outdoor Bike took place during the minutes 572 through 584. Example 2: Observed Walk spanning minutes 402 to 408. An instance of Walk was identified from minute 652 to 663. A continuous Walk phase from minute 692 t...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.