Purpose: The intelligibility of an accent strongly depends on the specific talker-listener pairing. To explore the causes of this phenomenon, we investigated the relationship between acoustic-phonetic similarity and accent intelligibility across native (1st language) and nonnative (2nd language) talker-listener pairings. We also used online measures to observe processing differences in quiet. Method: English (n = 16) and Spanish (n = 16) listeners heard Standard Southern British English, Glaswegian English, and Spanish-accented English in a speech recognition task (in quiet and noise) and an electroencephalogram task (quiet only) designed to assess phonological and lexical processing. Stimuli were drawn from the nonnative speech recognition sentences (Stringer & Iverson, 2019). The acoustic-phonetic similarity between listeners' accents and the 3 accents was calculated using the ACCDIST metric (Huckvale, 2004, 2007). Results: Talker-listener pairing had a clear influence on accent intelligibility. This was linked to the phonetic similarity of the talkers and the listeners, but similarity could not account for all findings. The influence of talker-listener pairing on lexical processing was less clear; the N400 effect was mostly robust to accent mismatches, with some relationship to intelligibility. Conclusion: These findings suggest that the influence of talker-listener pairing on intelligibility may be partly attributable to accent similarity in addition to accent familiarity. Online measures also show that differences in talker-listener accents can disrupt processing in quiet even where accents are highly intelligible.