To find out whether the education community shares a collective understanding about how students should be evaluated, we surveyed 202 educators (from all grade levels) and scientists attending assessment workshops (Pennsylvania, California, and Massachusetts) or judging a national student competition (Washington, DC). The educators and scientists graded hypothetical student responses to trivial math problems with definitive answers. Depending on the person grading the question, the same student answer received anywhere from no points to full credit. When the instructions preceding a question changed, the graders often changed how they evaluated the students, even though the evidence about what the student knew remained the same. After the instructions changed, some graders awarded more credit for three wrong answers than for three right answers. The graders shared no consensus about how student answers should be graded. If students are going to be evaluated using tests, the education community must create tighter rubrics that ensure a higher degree of inter- and intragrader reliability. (Contains 1 table and 1 figure.)