This article describes the design and evaluation of an online, asynchronous tool that mirrors the beneficial peer instruction process currently conducted in face-to-face classrooms. In the online Peer Instruction tool, students respond to a multiple-choice question with an answer and explanation. They are then exposed to peer responses to the same question. Finally, they may optionally submit a revised response, formed after exposure to these peer responses. Results from an experiment in three large-scale courses highlighted the perceived benefits for students who chose to engage meaningfully with the tool and showed opportunities for improving the user experience to increase engagement. After redesigning the interface and interaction, the original and revised versions of the tool were assessed in usability testing with students. The findings suggested the redesign is an improvement, reveal potential reasons the original design may have failed to engage more students, and reinforce the importance of understanding the multiple layers of the student experience when designing any educational technology.