Whenever classification decisions are made on educational tests, such as pass/fail, or basic, proficient, or advanced, the consistency and accuracy of those decisions should be estimated and reported. Methods for estimating the reliability of classification decisions made on the basis of educational tests are well-established (e.g., Rudner, 2001; Rudner, 2005; Lee, 2010). However, they are not covered in most measurement textbooks and so they are not widely known. Moreover, few practitioners are aware of freely available software that can be used to implement current methods for evaluating decision consistency and decision accuracy that are appropriate for contemporary educational assessments. In this article, we describe current methods for estimating decision consistency and decision accuracy and provide descriptions of "freeware" software that can estimate these statistics. Similarities and differences across these software packages are discussed. We focus on methods based on item response theory, which are particularly well-suited to most 21st century assessments.