A technique to estimate variances of weighted estimates in program evaluation is discussed. BRR estimates of variance are constructed from orthogonally weighted subsamples and can provide better estimates from complex samples than can other methods. Use of BRR is illustrated in a government evaluation of a food service program. (SLD)

Methods of probability modeling to analyze rater agreement are described, emphasizing their basic similarities and viewing them as variants of a common methodology. Statistical techniques for analyzing agreement data are described to address questions such as how many opinions are required to make a medical diagnosis with necessary accuracy. Kappa…

The following models which provide an estimate of the reliability of a rating scale are described: (1) disattenuation; (2) common factor; (3) external criterion; (4) treatment effects; and (5) regression. Such models are especially useful in selecting, evaluating, and training participants in clinical research. (SLD)

Statistical significance and "abnormality" have been used as criteria for the evaluation of intra-individual subtest score differences. Shortcomings of these criteria are identified, and improved estimates of the true score differences are suggested. The applicability of the abnormality criterion to these improved estimates is reviewed.…

A procedure for evaluating a variety of rater reliability models is presented. A multivariate linear model is used to describe and assess a set of ratings. Parameters are represented in terms of a factor analytic model, and maximum likelihood methods test the model parameters. Illustrative examples are presented. (SLD)

Studies of differential item functioning (DIF) under item response theory require that item parameter estimates be placed on the same metric before comparisons can be made. Evidence that methods for linking metrics may be influenced by the presence of differentially functioning items has been inconsistent. The effects of three methods for linking…

The effects of certain characteristics (e.g., sample design) of National Assessment of Educational Progress (NAEP) data on statistical analysis techniques are considered. Ignoring special features of NAEP data and proceeding with a standard analysis can produce inferences that underestimate the true variability and overestimate the true degrees of…

A general longitudinal model is presented for estimating school effects and their stability. The model, capable of separating true changes from sampling and measurement error, controls statistically for effects of factors exogenous to the school system. The model is illustrated with data from large cohorts of students in Scotland. (SLD)

Statistical methods for program evaluation with nonexperimental data are reviewed with emphasis on circumstances in which nonexperimental data are valid. Three solutions are proposed for problems of selection bias, and implications for evaluation design and data collection and analysis are discussed. (SLD)

Several goodness of fit (GOF) criteria have been developed to assist the researcher in interpreting structural equation models. However, the determination of GOF for structural equation models is not as straightforward as that for other statistical approaches in multivariate procedures. The four GOF criteria used across the commonly used…

Procedures for obtaining student samples for the National Assessment of Educational Progress (NAEP) and deriving survey weights for analysis of survey data are described. Sample designs are economically and operationally feasible, and weighting procedures result in increased precision of estimates as they account for the probabilities of student…

The effectiveness of alternative procedures to correct for rater leniency/stringency effects was studied when true scores were known. Ordinary least squares, weighted least squares, and imputation of the missing data consistently outperformed averaging the observed ratings; and the imputation technique was superior to the least squares methods.…

Methods for predicting specific college course grades, based on small numbers of observations, were investigated. These methods use collateral information across potentially diverse institutions to obtain refined within-group parameter estimates. One method, referred to as pooled least squares with adjusted intercepts, assumes that slopes and…

Six existing statistical procedures (models) for indirectly assessing the prevalence of need for alcohol, drug abuse, and mental health services across subareas of a state are described and critiqued in terms of epidemiological and statistical characteristics. Specific advantages and disadvantages are listed for each model. (SLD)

Two methods of using collateral information from similar institutions to predict college freshman grade average were investigated. One central prediction model, referred to as pooled least squares with adjusted intercepts, assumes that slopes and residual variances are homogeneous across selected colleges. The second model, referred to as Bayesian…