To set an optimal decision criterion on a test of recognition, a subject must estimate the degree to which they can discriminate previously studied from unstudied stimuli. To do so accurately, the subject must assess not only their mastery of the material but also the extent to which the distractors yield mnemonic evidence that makes them difficult to discriminate from studied items. In these experiments, we manipulated the degree to which the distractor set overlapped semantically or categorically with the previously learned stimuli, and examined the effects on criterion placement. When plausibility was manipulated by semantic membership, subjects who took tests with more plausible distractors set a higher criterion; for all other manipulations of plausibility, there were no between-subject effects of plausibility on criterion placement. However, over multiple test opportunities, subjects increased their criterion when the tests became more difficult, but they did not lower their criterion when the test became easier. This asymmetry obtained with picture and word categories, and suggests that online monitoring of recognition performance modulates the criterion shift.