Language testing is witnessing increasing interest in the potential for AI to support test development and validation. To date, published research involving AI in language testing has typically been conducted in the context of high-stakes proficiency tests and the potential for this technology to support local language testing is under-researched. The current study seeks to address this gap by reporting on the piloting of an AI generated language test in the context of a university in the UK. The focus of the paper was selected to explore two key areas in the work of J.D. Brown, namely pragmatics and connected speech. In the study, international students with English as a second language completed an AI generated test of pragmatics created from a transcript of spontaneous interaction, and a test-taking strategy questionnaire. Results demonstrated that the test did not provide a sufficiently reliable measurement of test takers' pragmatic competence, although reliability estimates did improve with the removal of problematic items. Implications for language test development involving AI are discussed.