While generative artificial intelligence (AI) tools are gradually being integrated into educational practice, their actual usability in classroom settings remains insufficiently understood. This mixed-methods research was designed as a small-sample pilot study, offering preliminary insights to inform a future large-scale scientific evaluation. The following questions were explored: What are the contents, functions, and capabilities of AI-based chatbots in education? How do students evaluate the usability of a chatbot using BUS-15 metrics? The online developer system botpress.com was chosen as the framework to support physics education within STEM fields for upper-secondary students. A validated instrument, BUS-15, a commercial tool for measuring chatbot usability, was utilised in an educational setting to evaluate students' user experience. In May 2025, 37 Taiwanese students used the developed AI chatbot. Cronbach's [alpha] was 0.709, consistent with values typically observed for chatbots developed for business purposes. Based on the results, the chatbot met the students' expectations. Accessibility emerged as a key determinant of chatbot success, although issues were noted with response times and conversational coherence. The findings also underscore the importance of evaluating a chatbot's pedagogical value, specifically its potential to enhance learning experiences and support multilingual engagement.