Gamification, or the intentional use of gaming elements in non-game contexts, has been touted as a promising tool to improve educational outcomes in online education, yet the evidence regarding why it might work and its effectiveness is inconclusive. One reason is that previous research has often included several gamification tools together, neglecting that each gamification tool can vary in effectiveness. In order to evaluate their relative impact, two frequently used gamification tools, badges (i.e., digital credentials given for achievements) and leaderboards (i.e., digital rankings based on performance), were compared for their effectiveness on the academic performance and motivation of students. Two experiments were conducted in two online undergraduate physics courses taught via a learning management system. In Experiment 1 (N = 102), badges and leaderboards were implemented in only one part of the course grading system (i.e., quizzes). In Experiment 2 (N = 88), all course grading system was gamified (i.e., quizzes and assignments). Four groups were created by random assignment of participants: badges-only, leaderboards-only, badges with leaderboards, and control (i.e., no badges, no leaderboards). Academic performance was measured by comparing quiz scores among groups in Experiment 1 and both quiz and assignment scores in Experiment 2. Participants filled out a self-report motivation survey about badges and leaderboards at the end of the study. Two experiments yielded similar results: badges and leaderboards did not affect participants' academic performance; however, most students approached them positively as motivational tools and wanted to see them in future online classes.