A pilot study on the development and administration of a test using a spatial reasoning problem, the 15-puzzle, is described. The test utilizes on-line capabilities of a real-time computer to record an examinee's progress on each problem through a sequence of problem-solving "moves", and to collect additional on-line data that might be of relevance to the evaluation of examinee performance. The examinees, 61 students in an introductory psychology class, were required to type a sequence of moves that would bring an array of scrambled numbers (start configuration) into agreement with a second array (goal configuration), using as few moves as possible. Results suggest that four performance indices might be useful in indexing problem difficulty: (1) mean number of moves in the sample; (2) proportion of students solving the problem; (3) proportion of students solving the problem in the optimal number of moves; and (4) a Special Difficulty Index, defined as the sample mean number of moves divided by the minimum number of moves required. Additional methods of scoring test and individual problem performance are studied. (Author/RL)