In a number of settings risk prediction models are being used to predict distal future outcomes for individuals, including high school risk prediction. We propose a new method, non-overlapping-leave-future-out (NOLFO) validation, to be used in settings with long delays between feature and outcome observation and where there are overlapping cohorts. Using NOLFO validation prevents temporal information leakage between the training and test sets. We apply this method to high school risk prediction, using data from a large-scale platform, and find that models are able to maintain their accuracy over long periods of time when tested on fully unseen data in most cases. These findings imply that organizations may be able to reduce the frequency of model retraining without sacrificing accuracy. In contexts such as long-term at-risk prediction with overlapping cohorts and long delays between feature and outcome observation, NOLFO is an important tool for ensuring that estimated model accuracy is representative of what can be expected in implementation. [For the complete proceedings, see ED675485.]