Purpose: The Comprehensive Assessment of Leadership for Learning (CALL) is an online task-based assessment of distributed instructional leadership. In developing CALL, researchers faced the challenge of structuring survey items that would measure leadership practice rather than individual traits. Critical in this work was developing items that accurately reflected current leadership practice. Therefore, the purpose of this paper is to pilot the CALL instrument and conducted cognitive validity testing on the instrument. Design/methodology/approach: CALL researchers piloted the survey in six schools in Wisconsin in order to test and refine the survey instrument. Researchers conducted cognitive walk-through interviews with five participants from each school: principals, associate principals, teachers, department chairs, guidance counselors, and activities directors. The interviews focussed on specific items in order to observe the users' thought processes and rationale in choosing a response for each item. The researchers focussed on relevancy, clarity, and accuracy of survey items in collecting and analyzing the resulting data. Findings: Three specific survey items were identified that exemplify these challenges and opportunities such as: accessible language, extended leadership, socially desirable responding, 360-degree perspectives, applying appropriate terminology, and identifying appropriate practices. These findings provided insight into survey development work and implications of distributed leadership. The authors discuss the challenges of creating a task-based leadership assessment. Originality/value: Developing a formative assessment of school leadership is valuable in supporting school leaders' work. The process of utilizing a qualitative approach to develop a quantitative instrument has proven critical in measuring task-based distributed leadership.