Attention control abilities are relevant for learning success. Little is known about the development of audio-visual attention in early childhood. Four groups of children between the ages of 4 and 10 years and adults performed an audio-visual distraction paradigm (N = 106). Multilevel analyses revealed increased reaction times in a visual categorization task when task-irrelevant novel sounds were presented, demonstrating involuntary distraction of attention. This distraction effect decreased with age and significantly differed between age groups. In addition, the two youngest age groups responded with a delay in trials following a distractor trial, indicating delayed reallocation of attention to the task at hand. Results indicate a significant maturation of audio-visual attention control within a few years during early childhood that continues throughout middle childhood.