Visual demands associated with in-vehicle display usage and text messaging distract a driver’s visual attention from the roadway. To minimize eyes-off-the-road demands, voice interaction systems are widely introduced. Under cognitively distracted condition, however, awareness of the operating environment will be degraded although the driver remains oriented to the roadway. It is also know that the risk of inattentive driving varies with age, thus systematic analysis of driving risks is required for the older drivers. This paper aims to understand the age-related driving performance degradation and visual attention changes under auditory cognitive demand which consists of three graded levels of cognitive complexity. In this study, two groups, aged 25-35 and 60-69, engaged in a delayed auditory recall task, so called N-back task, while driving a simulated highway. Comparisons of younger and older drivers’ driving performance including mean speed, speed variability and standard deviation of lane position, and gaze dispersion changes, which consist of x-axis and y-axis of visual attention, were conducted. As a result, it was observed that gaze dispersion decreased with each level of demand, demonstrating that these indices can correctly rank order cognitive workload. Moreover, gaze dispersion change patterns were quite consistent in younger and older age groups. Effects were also observed on driving performance measures, but they were subtle, nonlinear, and did not effectively differentiate the levels of cognitive workload.