In order to detect vehicles on the road reliably, a vehicle detector and tracker should be integrated to work in unison. In real applications, some of the ROIs generated from a vehicle detector are often ill-fitting due to imperfect detector outputs. The ill-fitting ROIs make it difficult for tracker to estimate a target vehicle correctly due to outliers. In this paper, we propose a stereo-based visual tracking method using a 3D feature clustering scheme to overcome this problem. Our method selects reliable features using feature matching and a 3D feature clustering method and estimates an accurate transform model using a modified RANSAC algorithm. Our experimental results demonstrate that the proposed method offers better performance compared with previous feature-based tracking methods.