Recent evidence has shown that eliciting confidence ratings can affect cognitive performance--a so-called reactivity effect. Several mechanisms have been proposed to account for reactivity, but currently there is only indirect evidence about why confidence ratings are reactive. Here, we explore the strategic changes in cognitive processes that occur in response to confidence ratings. Using a category learning paradigm that distinguishes between memorization and rule learning, in a large-sample (N = 710) pre-registered study, we show that eliciting confidence ratings caused a reduction in rule-based learning, even when compared to participants who made judgments of learning that asked them to reflect on their learning during the task as a whole. We argue that confidence ratings promote changes in the strategies participants adopt, driven by a more conservative approach that prioritizes performance over mastery and ultimately hinders rule-based learning and knowledge transfer.