Past studies show that novel auditory stimuli, presented in the context of an otherwise repeated sound, capture participants' attention away from a focal task, resulting in measurable behavioral distraction. Novel sounds are traditionally defined as rare and unexpected but past studies have not sought to disentangle these concepts directly. Using a cross-modal oddball task, we contrasted these aspects orthogonally by manipulating the base rate and conditional probabilities of sound events. We report for the first time that behavioral distraction does not result from a sound's novelty "per se" but from the violation of the cognitive system's expectation based on the learning of conditional probabilities and, to some extent, the occurrence of a perceptual change from one sound to another.