Remote heart rate (rHR) estimation, which aims to measure heart activities without any physical contact with the subject, is performed using remote photoplethys- mography (rPPG) and has great potential in many applications. In this paper, we introduce a remote heart rate estimation algorithm to detect driver drowsiness. The proposed method consists of two parts: an rPPGNet for PPG signal prediction from input video frames and an rHRNet for heart rate estimation from predicted PPG signal. Moreover, we apply a skin-based attention module and partition constraint to estimate more accurate rHR. To evaluate the performance of the proposed network, we train and test the proposed network on three public datasets.