We investigate two non-iterative estimation procedures for Rasch models, the pair-wise estimation procedure (PAIR) and the Eigenvector method (EVM), and identify theoretical issues with EVM for rating scale model (RSM) threshold estimation. We develop a new procedure to resolve these issues--the conditional pairwise adjacent thresholds procedure…

To detect differential item functioning (DIF), Rasch trees search for optimal split-points in covariates and identify subgroups of respondents in a data-driven way. To determine whether and in which covariate a split should be performed, Rasch trees use statistical significance tests. Consequently, Rasch trees are more likely to label small DIF…

This study adapted an effect size measure used for studying differential item functioning (DIF) in unidimensional tests and extended the measure to multidimensional tests. Two effect size measures were considered in a multidimensional item response theory model: signed weighted P-difference and unsigned weighted P-difference. The performance of…

Many education research studies employ small samples, which in turn lowers statistical power. We re-analyzed the results of a meta-analysis of simulation-based education to determine study power across a range of effect sizes, and the smallest effect that could be plausibly excluded. We systematically searched multiple databases through May 2011,…

In spite of the growing interest in cross-cultural research and assessment, there is little research on statistical procedures that can be used to simultaneously assess the differential item functioning (DIF) across multiple groups. The chief objective of this work is to show a unified framework for the analysis of DIF in multiple groups using one…

This study is an evaluation of the behavior of the Liu-Agresti estimator of the cumulative common odds ratio when identifying differential item functioning (DIF) with polytomously scored test items using small samples. The Liu-Agresti estimator has been proposed by Penfield and Algina as a promising approach for the study of polytomous DIF but no…

Designing for the needs of people with impairments has rarely been a significant feature of urban planning theory and education. Given the role of urban planners as shapers of the built environment and public policy, the prevalence of negative and misinformed attitudes among planners toward impaired populations has been highlighted as requiring…

Standardized testing has been part of the American educational system for decades. Controversy from the beginning has plagued standardized testing, is plaguing testing today, and will continue to be controversial. Given the current federal educational policies supporting increased standardized testing, psychometricians, educators and policy makers…

Confidence intervals are widely accepted as a preferred way to present study results. They encompass significance tests and provide an estimate of the magnitude of the effect. However, comparisons of correlations still rely heavily on significance testing. The persistence of this practice is caused primarily by the lack of simple yet accurate…

A recent survey of simulation studies concluded that an overwhelming majority of papers do not report a rationale for the number of iterations carried out in Monte Carlo robustness (MCR) experiments. The survey suggested that researchers might benefit from adopting a hypothesis testing strategy in the planning and reporting of simulation studies.…

The level of standardized effect sizes obtained by chance and the use of significance tests to guard against spuriously high standardized effect sizes were studied. The concept of the "protected effect size" is also introduced. Monte Carlo methods were used to generate data for the study using random normal deviates as the basis for sample means…

Whether one should use null hypothesis testing, confidence intervals, and/or effect sizes is a source of continuing controversy in educational research. An alternative to testing for statistical significance, known as equivalence testing, is little used in educational research. Equivalence testing is useful in situations where the researcher…

This paper suggests that statistical significance testing and effect size are two sides of the same coin; they complement each other, but do not substitute for one another. Good research practice requires that both should be taken into consideration to make sound quantitative decisions. A Monte Carlo simulation experiment was conducted, and a…

The use of p-values in combining results of studies often involves studies that are potentially aberrant. This paper proposes a combined test that permits trimming some of the extreme p-values. The trimmed statistic is based on an inverse cumulative normal transformation of the ordered p-values. (SLD)

Formulae for estimating individual rater reliabilities from analysis of treatment effects are presented and evaluated. Monte Carlo methods illustrate the formulae. Results indicate that large sample sizes, large true treatment effects, and large differences in the actual reliabilities of raters are required for the approach to be useful. (SLD)