Behavior rating scales measuring executive function have grown popular among school psychologists and other professions. To examine the generalizability of executive function rating scales, 42 parent-adolescent dyads, recruited from a school-based sample, completed the Behavior Rating Inventory of Executive Function, Second Edition and the Comprehensive Executive Function Inventory. Caregivers were mostly mothers (79%), and adolescents were mostly girls (81%). Both groups included approximately two-thirds participants who were White and not-Hispanic, and more than a quarter of participants were Black and not-Hispanic. A total of 1260 scores representing general and specific executive functions were submitted to generalizability theory analyses to evaluate rater effects, instrument effects, dimension effects, and all interactions on the variation across the total scores and subscale scores from the rating scales. The generalizability of scores was markedly low (0.09-0.19) and much lower than expected given prior research and analysis evaluating these effects in isolation. Partial models examining variance components for each rater in isolation indicated higher yet inadequate generalizability (0.59-0.69). These findings indicate that practitioners should be cautious when generalizing ratings of executive function across different raters, similar instruments, and specific executive functions--especially in school-based populations like those from which we sampled.