Although exploratory play is considered a hallmark of cognitive development and learning, relatively few studies have been able to quantitatively characterize the shifts that may occur in children's approach to exploration. One reason for this gap is due to challenges coding and analyzing children's exploratory play behavior. In our paper, we employ a novel computational modeling approach to understand whether and how children's exploratory play patterns shift in early childhood (3- to 11-years-old). We analyze data from children (N = 432) across five different experiments that varied in the type of exploration task (including novel toys, novel topics, and novel envelopes). Children's behaviors were coded action-by-action according to whether children repeated an action on the same type of target, switched to a novel target, or terminated play. Our computational Markov model searches over the space of possible "stay," "switch," and "end" parameters to quantify child-specific transition probabilities. We find that overall, older children are less likely to perseverate, more likely to switch, and more likely to end the task earlier. Our approach provides a demonstration of how Markov models can be used to map the process of play, providing insight into theories of developmental changes in exploration.