The reliability and validity of course evaluations in higher education is often assumed. The typical Likert-type surveys utilized when students' evaluate the course and instructor often overlook measurement issues, or deal with them in an ineffective manner. Given the importance that is placed on higher education course evaluations, with results impacting such events as merit raises and promotion, the proper construction and use of evaluation tools is a critical issue. In an effort to assure "honesty" in student responses, many institutions include items written positively and negatively, which are intended to measure the same construct. Using 537 course evaluations for a mathematics faculty member at a Midwest college, an item analysis is conducted with attention given to means and standard deviations, frequency counts, nonparametric correlations and tests of significant differences between questions that should, in theory, produce a similar measure or exactly opposite. A contention is made that the way the item is asked does matter, at least in some instances, and it should not be assumed that an item written in the positive and negative should directly correlate. The survey research community and institutions utilizing similar rating scale instruments will benefit from the results of this study, as well as the education community in general. (Contains 5 tables and 2 figures.)