Most organic chemistry reactions occur by a mechanism that includes acid-base chemistry, so it is important that students develop and learn to use correct conceptions of acids and acid strength. Recent studies have described undergraduate organic chemistry students' cognitive resources related to the Brønsted-Lowry acid model and the Lewis acid model, providing both qualitative and quantitative analyses of these understandings. To drive changes in pedagogy and curriculum, however, faculty need to be able to quickly assess students' conceptions of acids and acid strength. We recently reported on the development and assessment of a nine-item, multiple-tier, multiple-choice concept inventory about acid strength, named ACID I. Coefficient a for ACID I was calculated to be below 0.70. In this manuscript, we demonstrate that despite this low coefficient "a", the data generated by ACID I are indeed reliable. Thus, the purpose of this paper is to (i) report two significant alternative conceptions about acid strength that persist in organic chemistry students' minds after nearly two semesters, and (ii) discuss the meaning of reliability for concept inventories, including a description of additional measures for the reliability of data collected using ACID I. Two types of test conditions were employed within second-semester organic chemistry courses in two different regions of the United States: a course at a medium-sized, midwestern liberal arts university and a large, southeastern research university.