This paper demonstrates that a meta-analysis technique applied to the Solomon Four-Group Design (SFGD) can fail to find significance even though an earlier "weaker" test may have found significance. The meta-analysis technique was promoted by Braver and Braver as the most powerful single test for analyzing data from an SFGD. They…

A generalized measure of association and an associated test of significance are presented for nominal independent variables in which any number or combination of interval, ordinal, or nominal dependent variables can be analyzed. A permutation test of significance is provided for the new measure. (SLD)

Different ways that dependent variables can cause multivariate significance are illustrated. It is argued that univariate tests should not be used as follow-up procedures to identify the variables that can contribute to multivariate significance. Methods that can be used to determine the contributions individual variables make to multivariate…

An Edgeworth approximation for accurate significance probabilities for the Wilcoxon two-sample test is substantially simplified. A method is developed that allows quick calculations of very accurate probabilities. Exact formulas are given for most of the remaining cases, and tables are presented comparing the new simplification to likely…

Large sample chi-square tests of the significance of the difference between two correlated kappas, weighted or unweighted, are derived. Cases are presented with one judge in common between the two kappas and no judge in common. An illustrative calculation is included. (Author/SLD)

Statistical significance is often misinterpreted to mean replicability or generalizability of results, although a statistically significant difference does not equal a reliable difference. Sample splitting procedures may be a more accurate way of estimating research result generalizability. This type of cross-validation involves randomly dividing…

A new coefficient--psi-star--is presented for measuring agreement on bidirectional response continua. The coefficient is designed to measure agreement in larger square contingency tables with even numbers of rows and columns. Advantages of the procedure are discussed. (SLD)

Use of "special" orthonormal mean contrasts and mean contrast variances can help educational researchers interpret a wide variety of repeated measures data. Most statistical packages allow educational researchers to test for differences across repeated measures using both the univariate mixed model F test and a multivariate test.…

The asymptotic sampling distributions (means and variances) are presented for the signed and unsigned estimates for the Rasch model, two-parameter model, and the three-parameter model with fixed lower asymptotes. Applications for item-bias research are discussed. (SLD)

The mean and variance of the statistic psi-star, which is useful for measuring agreement between two raters and allows for tests of significance in the case of large samples, are considered. Results may also be applied for the "G" index of J. W. Holley and J. P. Guilford (1964). (SLD)

Two coefficients ("H" and "C") for evaluating the homogeneity or congruence of a set of ratings or scores are described. Statistical tests of the significance of these indexes are presented, along with a description of a computer program (CHOPRO) for determining values of "C" and "H" and associated…

A brief overview is provided of the Conger-Lipshitz approach to estimating the reliability of a profile or test battery. A computational example from a recent study shows how canonical reliability can be obtained through existing statistical software. (SLD)

Some of the methods originally used to find relationships between reliability and power associated with a single measurement are extended to difference scores. Results, based on explicit power calculations, show that augmenting the reliability of measurement by reducing error score variance can make significance tests of difference more powerful.…

A method is proposed for calculating the sample size needed to achieve acceptable statistical power with a given test. The minimally important difference significant (MIDS) criterion for sample size is explained and supported with recommendations for determining sample size. The MIDS criterion is computationally simple and easy to explain. (SLD)

An extension is proposed for the network algorithm introduced by C.R. Mehta and N.R. Patel to construct exact tail probabilities for testing the general hypothesis that item responses are distributed according to the Rasch model. A simulation study indicates the efficiency of the algorithm. (SLD)