In this paper we show how we can utilize human-guided machine learning techniques coupled with a learning science practitioner interface (DataShop) to identify potential improvements to existing educational technology. Specifically, we provide an interface for the classification of underlying Knowledge Components (KCs) to better model student learning. The configurable interface allows users to quickly and accurately identify areas of improvement based on the analysis of learning curves. We present two cases where the interface and accompanying methods have been applied in the domains of geometry and psychology to improve upon existing student models. Both cases present outcomes of better models that more closely model student learning. We reflect on how to iterate upon the educational technology used for the respective courses based on these better models and further opportunities for utilizing the system to other domains, such as computing principles. [This paper was published in: "REV 2020, AISC 1231," edited by M. E. Auer and D. May, Springer Nature Switzerland, 2021, pp. 742-61.]