Facial masks have become and may remain ubiquitous. Though important for preventing infection, they may also serve as a reminder of the risks of disease. Thus, they may either act as cues for threat, priming avoidance-related behavior, or as cues for a safe interaction, priming social approach. To distinguish between these possibilities, we assessed implicit and explicit evaluations of masked individuals as well as avoidance bias toward relatively unsafe interactions with unmasked individuals in an approach-avoidance task in an online study. We further assessed COVID-19 anxiety and specific attitudes toward mask-wearing, including mask effectiveness and desirability, hindrance of communication from masks, aesthetic appeal of masks, and mask-related worrying. Across one sample of younger (18-35 years, N = 147) and one of older adults (60+ years, N = 150), we found neither an average approach nor avoidance bias toward mask-wearing compared to unmasked individuals in the indirect behavior measurement task. However, across the combined sample, self-reported mask-related worrying correlated with reduced avoidance tendencies toward unmasked individuals when COVID-19 anxiety was low, but not when it was high. This relationship was specific to avoidance tendencies and was not observed in respect to explicit or implicit preference for mask-wearing individuals. We conclude that unsafe interaction styles may be reduced by targeting mask-related worrying with public interventions, in particular for populations that otherwise have low generalized COVID-19 anxiety.