Objective: To establish the validity and reliability of an online assessment instrument's items developed to track educational outcomes over time. Design and Setting: A descriptive study of the validation arguments and reliability testing of the assessment items. The instrument is available to graduating students enrolled in entry-level Athletic Training Education Programs (ATEPs). Methods: Validity was established with the creation of a national advisory board of Athletic Training educators. Construct validity was established with the creation of a test blueprint to guide the development of items for the knowledge exam. Internal reliability estimates for each domain were calculated. A single scale reliability analysis was conducted using all items. An item analysis was conducted by calculating difficulty and discrimination indexes for each item. Results: The internal reliability estimates ranged from 0.23 to 0.44 suggesting that individual domain scores for this draft of the instrument were not reliable. The single scale total score reliability however, produced an alpha = 0.84 suggesting a high level of reliability. Difficulty index scores ranged from 0.03 to 0.99 (mean = 0.74 plus or minus 0.25). Discrimination index scores ranged from -0.01 to 0.41 (mean = 0.21 plus or minus 0.09). Conclusions: While the individual domain reliability was low, the overall single scale score is acceptable. Difficulty and discrimination index scores allowed the removal and revision of items to increase the overall reliability of the test bank. (Contains 2 figures.)