Young children excel at pragmatic inferences known as ad hoc quantity implicatures: they can infer, for example, that a speaker who said "the card with apples" meant the card with "nothing but" apples. However, it is not known whether children take into account the speaker's perspective in deriving such inferences, as adults are able to do, and as the received theories of pragmatics claim. In two experiments, we tested children (5-7 years, N = 33 and N = 25) and adults using a picture-matching director task, in which participants played a game giving cards to the speaker, with some cards being in common ground and some in privileged ground. We found that adults can both derive implicatures when all information is in common ground and not derive them when relevant information is in privileged ground. Children also derive ad hoc implicatures when relevant information is in common ground but, crucially, fail to not derive them when it is in privileged ground. Children's difficulty with integrating perspective-taking with pragmatic inferencing challenges the received theories about the necessity of perspective-taking in pragmatics.