Voluntary action can be studied by giving participants free choice over which task to perform in response to each presented stimulus. In such experiments, performance costs are observed when participants choose to switch tasks from the previous trial. It has been proposed that these costs primarily index the time-consuming operation of top-down control processes that support voluntary action. The present experiments showed, contrary to this view, that greater costs were associated with voluntary switching to the easier task of a pair. These increased switch costs for the easier task were accompanied by a reliable preference of the participants for performing the other, more difficult task. Interference between tasks during response selection was identified as the critical factor driving these effects of task difficulty. Together, the findings suggest that participants' voluntary choices, and the time taken to execute those choices, may not directly index the operation of cognitive control but instead may reflect complex interactions between top-down and bottom-up influences on behavior. (Contains 3 tables, 4 figures and 1 footnote.)