Multimodal Remote Heart Rate Estimation via Spatio-Temporal Transformers and Adaptive Fusion

Abstract

Remote photoplethysmography (rPPG) enables noncontact heart rate (HR) estimation from facial videos. Despite recent advances, single-modality methods remain vulnerable to motion, illumination changes, and modality-specific degradations. We address these limitations with a multimodal framework that explicitly leverages complementary RGB and infrared (IR, thermal or NIR) streams. Built on a 3-D SwiftFormer backbone, the method integrates three modules: 1) a context-aware temporal difference convolution (CTDC) that amplifies motion-sensitive cues via multiscale temporal differencing; 2) a bidirectional cross-attention (BCA) that enables hierarchical information exchange between modalities; and 3) a cross-modal gating fusion (CMGF) that adaptively combines features using a temperature-scaled logit-difference gate. Training is guided by a hybrid objective over time and frequency, augmented with a scheduled soft-DTW alignment term. Extensive experiments on two public datasets demonstrate consistent improvements over state-of-the-art baselines, with ablation studies confirming the contributions of CTDC, BCA, CMGF, and soft-DTW. These results highlight the effectiveness of explicit cross-modal interaction and adaptive fusion for robust, accurate remote HR (rHR) estimation.