This report describes the design of a test of one aspect of critical thinking ability, the ability to correctly appraise observations. Intended for classroom use with senior high school students, the 50 item Test on Appraising Observations is based on a comprehensive set of principles modified from Robert Ennes' conception of good observation appraisal. The test evolved through many versions using methodology developed from the construct validity theory that ability tests are valid to the extent that good thinking leads to good test performance and that poor thinking leads to bad test performance. The methodology involved the systematic collection of thinking about protocols of examinees while they worked through test questions. Data collection and analysis for two experimental test versions are described in detail. Item and test statistics for the final version were collected from four southern Ontario high schools and compared to results of two other critical thinking tests. The appendices contain two versions of the Test on Appraising Observations; the Test on Assessing the Believability of Observation Statements; the Observation Test Interview Model, B; the Instruction Sheet to Cooperating Teachers; and the answer key and principles tested per item. (BS)