Three common methodological problems in the analysis of observational data include: (1) failure to base degrees of freedom for the analysis on the appropriate sampling unit; (2) the validity of the design and data are inconsistent with the generalizations reached; (3) the statistic used to calculate coder reliability is often inappropriate and yields values that are misleading. A simulated study on the effects of observational training on teacher performance is employed to illustrate these problems. Hypotheses, research procedure, observer reliability, results, data analysis, and conclusions are included in the simulated report, which is criticized in terms of the problems outlined. (AE)