Abstract

The growing use of wearable devices capturing multimodal signals, such as electrocardiogram (ECG) and accelerometer (ACC) data, offers opportunities for creating large, real-life datasets that can power self-supervised learning (SSL) models that can support a variety of applications. Using ECG and ACC wearables data, we train SSL models with both unimodal and cross-modal pretraining objectives and evaluate their performance on downstream tasks such as physical activity recognition and affect classification. In each of our downstream experiments, we identify dominant and secondary modalities and demonstrate that cross-modal pretraining can enhance downstream performance even when only a single modality is available. Additionally, we find that modalities can complement or contradict each other in certain cases, highlighting the importance of incorporating domain knowledge when designing …

Date

2026

Authors

Dominika Kunc, Joanna Komoszyńska, Kleanthis Avramidis, Tiantian Feng, Shrikanth Narayanan, Przemysław Kazienko, Stanisław Saganowski

Journal

Information Fusion

Pages

104439

Publisher

Elsevier