Educational reform priorities such as emphasis on quantitative modelling (QM) have positioned undergraduate biology instructors as designers of QM experiences to engage students in authentic science practices that support the development of data-driven and evidence-based reasoning. Yet, little is known about how biology instructors adapt to the…

The present study examined the long-term usefulness of estimated parameters used to adjust the scores from a performance assessment to account for differences in rater stringency. Ratings from four components of the USMLE[R] Step 2 Clinical Skills Examination data were analyzed. A generalizability-theory framework was used to examine the extent to…

Cross-validation, an economical method for assessing whether sample results will generalize, is discussed in this paper. Cross-validation is an invariance technique that uses two subsets of the data sample to derive discriminant function coefficients. The two sets of coefficients are then used with each data subset to derive discriminant function…

A family of coefficients of relational agreement for numerical scales is proposed, generalizing to multiple judges the theory of F. E. Zegers and J. M. F. ten Berge (1985) of association coefficients for two variables, using the premise that choice of coefficient depends on scale type of the variables. (SLD)

A new method is presented for determining the minimum number of observations per subject needed to achieve a specific generalizability coefficient. The method, consisting of a branch-and-bound algorithm, enables an investigator to specify an acceptable threshold for generalizability coefficients. (SLD)

Multivariate methods are being used with increasing frequency in educational research because these methods control "experimentwise" error rate inflation, and because the methods best honor the nature of the reality to which the researcher wishes to generalize. This paper: explains the basic logic of canonical analysis; illustrates that…

The probability integral of the multivariate normal distribution (ND) has received considerable attention since W. F. Sheppard's (1900) and K. Pearson's (1901) seminal work on the bivariate ND. This paper evaluates the formula that represents the "n x n" correlation matrix of the "chi(sub i)" and the standardized multivariate…

A 2-facet measurement model is identified, and its coefficient of generalizability (CG) is examined. Three other multifaceted measurement models and their CGs are identified. An empirical investigation of all four procedures is conducted using data from a study of the psychopathology of 71 prison inmates. (SLD)

A methodology for determining the optimal number of observations to use in a measurement design when resource constraints are imposed is presented. Two- and three-facet designs are outlined. Parallel closed form formulae can easily be determined for other designs. (TJH)

A natural parameterization and formalization of the problem of measuring change in dichotomous data is developed. Mathematically-exact definitions of specific objectivity are presented, and the basic structures of the linear logistic test model and the linear logistic model with relaxed assumptions are clarified. (SLD)

The general transformation approach to time series analysis is extended to the analysis of multiple unit data by the development of a patterned transformation matrix. The procedure includes alternatives for special cases and requires only minor revisions in existing computer software. (SLD)

Demonstrates how perspectives from covariance structural modeling and generalizability theory can be combined for a comprehensive assessment of latent constructs. This approach to examining variance components is illustrated by one- and two- facet designs, and can be extended to more complex designs. (MAK)

In the process of developing a conditionally-dependent item response theory (IRT) model, the problem arose of modeling an underlying multivariate normal (MVN) response process with general correlation among the items. Without the assumption of conditional independence, for which the underlying MVN cdf takes on comparatively simple forms and can be…

The use of item response theory models for analyzing the results of judgmental standard setting studies (the Angoff technique) for establishing minimum pass levels is discussed. A comparison of three methods indicates the traditional approach may not be best. A procedure based on generalizability theory is suggested. (GDC)

An efficient search procedure is presented for determining the optimal number of observations of facets in a design that maximize generalizability when resource constraints are imposed. The procedure is illustrated for three-facet and four-facet designs, with extensions for other configurations. (Author/SLD)