The performance decision tree (PDT; Fulcher et al., 2011) is a rubric style that is applicable to performance assessment, with origins in Upshur and Turner's (1995) empirically derived binary-choice, boundary-definition (EBB) scale. It is easier for raters to assess performance by evaluating multiple binary-choice descriptors. Additionally, learners receive easy-to-understand feedback. The most important advantage is that the number of feedback types can be increased to the nth power of 2 (n = number of items). Therefore, this study compared the validity, reliability, and practicality of the PDT and grid-style rubrics. Three Japanese university English instructors participated in the preliminary work to create the rubric, and the contributors scrutinized the information to create a pilot version. Five raters (two English lecturers and three student raters) then scored 64 Japanese university students' performances in four English-speaking tasks. Next, analyses were conducted using the many-facet Rasch measurement and generalizability theory to observe the validity and reliability of the PDT. The results confirmed that both the PDT and the grid had validity within the range that could be examined and that the reliability was sufficiently high. However, interviews with raters revealed that the PDT had superior practicality. Additionally, the PDT took less time to complete the evaluation than the grid and was judged to be an effective form of the rubric. This article is part of a study aimed at demonstrating the usefulness of the PDT by creating and evaluating a pilot version. It was used to partially examine the validity and reliability of the PDT. The remaining aspects of validation shall be presented in a separate article.