Objective: The Coordinators of Psychiatric Education (COPE) Residency In-Training Exam is a formative exam for Canadian psychiatric residents that was reconstructed using assessment best practices. An assessment of psychometric properties was subsequently performed on the exam to ensure preliminary validity and reliability. Methods: An exam blueprint was developed based on the 2007 Royal College objectives for psychiatric training. A minimum pass level was established using a modified Angoff method. The exam was administered to all Canadian psychiatric residents in postgraduate years 2 to 5 with test reliability (Cronbach alpha) and item analysis performed. Exam validity was assessed through blueprint adherence and cross-year resident performance analysis. Results: Four hundred two exams were suitable for analysis. The overall mean score for all residents was 69.6% (SD = 8.5) with significant differences in total scores between each of the postgraduate year groups, with consistently better performance with increasing time in residency. Cronbach alpha was 0.79. Conclusion: The present study provides preliminary support that the reconstructed COPE Residency In-Training Exam demonstrates adequate reliability and validity, including showing the capacity to discriminate between levels of training.