This study presents a robust self-cancellation-based time-of-arrival (TOA) estimation algorithm against carrier frequency offset (CFO). The proposed algorithm is developed to exploit the up and down chirp of bi-directional chirp signals. Using the relationship between the phase shifts of the transformed signals of the up and down chirp symbols, the CFO-induced error of the TOA estimates can be cancelled without pre-estimation and compensation through the proposed method. The root mean squared error of the proposed algorithm is analysed and verified through simulations in an additive white Gaussian noise and multi-path channel in the presence of CFO.