The reliability of data is a critical issue in decision-making for practitioners in the school. Percent Agreement and Cohen's kappa are the two most widely reported indices of inter-rater reliability, however, a recent Monte Carlo study on the reliability of multi-category scales found other indices to be more trustworthy given the type of data and number of categories. This manuscript presents defensible decision steps, methods, and rationale for selecting and calculating inter-rater reliability for practitioners. In addition to using screen shots from readily available online programs for calculation such as Excel and Vassar College (Lowry, 2010), decision guides and a flow chart are presented in several figures and tables for easy reference. (Contains 4 tables and 7 figures.)