We describe a computer assisted language learning (CALL) system that uses human-machine dialogue as its medium of interaction. The system was developed to help students learn the basics of the Maori language and was designed to accompany the introductory course in Maori running at the University of Otago. The student engages in a task-based conversation with the CALL system, which is designed to provide practice in specific language constructions as well as to provide opportunities to negotiate meaning within the constraints of the target language constructions. The focus of the paper is on evaluating the CALL system with students. We began with a series of observations of a small number of students using an early version of the system. These real-use observations informed the development of the system which was used in a quantitative study with a different student cohort later in the year. Learning outcomes were assessed through written tests on the Maori system of personal pronouns. Our results show that students who attended the CALL tutorials scored at least as well as students who attended regular tutorials on immediate post-tests. The CALL tutorial groups scored less well on delayed post-tests and some possible reasons for this are discussed. Some caveats for the study are also discussed along with suggestions for further research. (Contains 2 tables and 3 figures.)