The reliability, factor structure, and discriminatory power of faculty perceptions of the performance of departmental administrators were investigated, using the Administrator Evaluation Survey (AES) developed at the University of Illinois. Faculty from a major research university in the midwest, a smaller state university in the midwest, and a predominantly black state university located in the southeast participated. The approximately 350 faculty respondents represented 16 departments. Factor loadings on the 17 items of the AES were determined for each university. Two functions appeared to separate the three universities in terms of (1) the head's departmental leadership--communicating departmental needs to higher level administrators, allocating resources among faculty, directing promotion and tenure process, and making difficult decisions; and (2) the head's departmental management--handling routine duties, resolving tensions, encouraging good teaching, making difficult decisions, and improving scholarly reputation. To determine whether the AES was able to detect differences among departments within a given institution, similar multivariate analyses were conducted on the major research university's ratings. Three functions appeared to separate the department heads in terms of the department head's contribution toward improving the reputation of the department, the head's administrative ability, and the head's handling of curricular/instructional matters. A revision of the AES to 11 items was also tested. Overall, it is concluded that factor analysis of the AES suggests that department administrators are evaluated by their faculty along one general dimension or factor. Possible reasons for the observed unidimensionality of the AES are considered. A sample questionnaire is included. (SW)