Purpose: The purpose of the current study was to assess dual-task interference (i.e., changes between the dual-task and baseline condition) in a listening effort dual-task paradigm in normal-hearing (NH) adults, hearing aid (HA) users, and cochlear implant (CI) users. Method: Three groups of 31 participants were included: (a) NH adults, (b) HA users, and (c) CI users. The dual-task paradigm consisted of a primary speech understanding task in a quiet condition, and a favorable and unfavorable noise condition, and a secondary visual memory task. Dual-task interference was calculated for both tasks, and participants were classified based on their patterns of interference. Descriptive analyses were established and differences between the three groups were examined. Results: The descriptive results showed varying patterns of dual-task interference between the three listening conditions. Most participants showed the pattern of visual memory interference (i.e., worse results for the secondary task in the dual-task condition and no difference for the primary task) in the quiet condition, whereas the pattern of speech understanding priority trade-off (i.e., worse results for the secondary task in the dual-task condition and better results for the primary task) was most prominent in the unfavorable noise condition. Particularly, in HA and CI users, this shift was seen. However, the patterns of dual-task interference were not statistically different between the three groups. Conclusions: Results of this study may provide additional insight into the interpretation of dual-task paradigms for measuring listening effort in diverse participant groups. It highlights the importance of considering both the primary and secondary tasks for accurate interpretation of results.