Models of attentional control usually describe online shifts in control settings that accommodate changing task demands. The current studies suggest that online control over distractor exclusion--a core component of visual selection--can be accomplished without online shifts in top-down settings. Measurements of target discrimination accuracy suggested that the degree of distractor exclusion was guided by retinotopic maps of the prior probability of distractor interference at the attended locations. These probability maps can be retrieved via object-based cues, and they interact with shifts of attention to elicit increased levels of distractor exclusion when it is most needed. Thus, static probability maps can provide an internal template that guides the resolution of visual interference as spatial attention traverses the visual field.