Assessing students' conceptions related to independence of events and determining probabilities from a sample space has been the focus of research in probability education for over 40 years. While we know a lot from past studies about predictable ways students may reason with well-known tasks, developing a diagnostic assessment that can be used by teachers to inform instruction demands the use of familiar and unfamiliar contexts. This paper presents the current work of a research team whose aim is to create a formative concept inventory with strong evidence of validity that uses a psychometric model to confidently predict whether a student exhibits one or more misconception across many items. We illustrate this process in this paper using a particular item with a context of a raffle aimed to measure whether a student reasons with misconceptions related to independence or equiprobability. The results of two aspects of the validity process: cognitive interviews to assess response proce sses on individual items, and a large-scale administration to examine internal structure of the concept inventory revealed difficulties in assessing students' reasoning about these key probability concepts and trends in the prevalence of misconceptions across grades. Results can provide guidance for others aiming to develop assessments in mathematics education and also support further possibilities for research into understanding students' reasoning about independence and sample space.