Contemporary case studies rely on verbal arguments and set theory to build or evaluate theoretical claims. While existing procedures excel in the use of qualitative information (information about kind), they ignore quantitative information (information about degree) at central points of the analysis. Effectively, contemporary case studies rely on…

Studies of group differences in intelligence often invite conclusions about groups in general from studies of group differences in selected populations. The same design is used in the study of group differences in other traits as well. Investigators observe samples from two groups (e.g. men and women) in some accessible population, but seek to…

The effects of using different data analysis methods on estimates of treatment effects of educational programs were investigated. Various regression models, such as those recommended for Title I program evaluations, were studied. The first effect studied was the amount of bias that might be expected to occur in the various settings. Results…

Model identification of time-series data is essential to valid statistical tests of intervention effects. Model identification is, at best, inexact in the social and behavioral sciences where one is often confronted with small numbers of observations. These problems are discussed, and the results of independent identifications of 130 social and…

The validity of the equipercentile hypothesis of the Title I Evaluation and Reporting System norm-referenced evaluation model was examined using 3,224 seventh- and ninth-grade students. Findings from confidence interval procedures contradicted the equipercentile hypothesis. There was a pattern of large gains for students not receiving any special…

Statistical methods for program evaluation with nonexperimental data are reviewed with emphasis on circumstances in which nonexperimental data are valid. Three solutions are proposed for problems of selection bias, and implications for evaluation design and data collection and analysis are discussed. (SLD)

Two related problems are discussed: developing a methodology which (1) will not require full commitment of a local education agency to new techniques which may be ineffective in local settings and (2) will allow graceful abandonment of a new technology that may be proved ineffective. An example is discussed. (CTM)

The norm-referenced evaluation model (RMC Model A) for Title I project evaluation, consists of procedures whereby the expected posttest standing of a treatment group under the null condition is generated from their pretest standing. It is assumed that the treatment group is not selected on the basis of their pretest scores and can be considered…

Title I evaluations using the RMC Model C design depend for their interpretation on the assumption that the regression of posttest on pretest is linear across the cut score level when there is no treatment; but there are many instances where nonlinearities may occur. If one applies the analysis of covariance, or model C analysis, large errors may…

Evaluations of social programs based upon quasi-experimental designs are typically plagued by problems of nonequivalence between the experimental and comparison group prior to the experiment. In such settings it is extremely difficult, if not impossible, to isolate the effects of the program from the confounding effects associated with the…

Research efforts have established that the Mantel-Haenszel procedure (MHP) is an effective method for detecting the presence of test items exhibiting differential item functioning (DIF). While the MHP has been advocated for situations where item response theory based methods may not be usable, recent findings have suggested that the performance of…

The three RMC models endorsed by the U.S. Office of Education for the evaluation of Elementary and Secondary Education Act Title I programs are based on narrowly conceived approaches to evaluation--the measurement of cognitive achievement gains. Each model requires the comparison of observed student performance with an estimate of what level of…