Detecting a horizontal line in an infrared image is an important component of automatic surveillance applications such as detecting ships, missiles on the horizon, unmanned aerial vehicle control, flight navigation, and port security. Most of the existing solutions for the problems only use single image to detect horizon line. Although this results in good accuracy for some images, it often fails to detect horizons in foggy, occluded environments. In this paper, we propose a novel horizon detection and tracking method that is robust to sensor vibrations and occlusions using an infrared camera for 24 hour running. An initial horizon is detected by a sensor geometry-based method for the robustness. Local horizon optimization and tracking produce stable horizons in occluded environments. The experimental results validate the feasibility of the proposed method in real infrared images.