The use of inter-rater reliability (IRR) methods may provide an opportunity to improve the transparency and consistency of qualitative case study data analysis in terms of the rigor of how codes and constructs have been developed from the raw data. Few articles on qualitative research methods in the literature conduct IRR assessments or neglect to…

The Grand Challenges Project supports global collaborations that inform equitable practices for assessment practitioners in higher education while identifying evidence-informed solutions. One goal named in the project is to leverage assessment findings to increase equity in higher education. This article focuses on the results from the Equity…

When good model-data fit is observed, the Many-Facet Rasch (MFR) model acts as a linking and equating model that can be used to estimate student achievement, item difficulties, and rater severity on the same linear continuum. Given sufficient connectivity among the facets, the MFR model provides estimates of student achievement that are equated to…

Responds to a critique by Mudford, Hogg, and Roberts (1997) that raised concerns about the observation code used in a longitudinal research project to assess emerging behavior state patterns in young children with disabilities. Concerns about the thoroughness of the reliability data collected by Mudford are discussed. (Author/CR)

In this rejoinder to John Chubb's reply to "Edison Is the Symptom, NCLB Is the Disease," the author argues that Edison offers feel-good measures without really solving any of the problem of schools in poverty. Defending his original argument, the author cites a RAND study that questions the results Chubb claims. The study indicates the…

The degree of agreement between two raters rating several objects for a single characteristic can be expressed through an association coefficient, such as the Pearson product-moment correlation. How to select an appropriate association coefficient, and the desirable properties and uses of a class of such coefficients--the Euclidean…

Science tends to find a solution to the problem of the unreliability of human perception by understanding objectivity as the absence of subjectivity. However, from a phenomenological point of view, subjectivity is not so much a problem as an inevitable starting-point. That does not mean that the problem of the correctness of people's accounts of…

Although single-subject research techniques are valuable to counselors, analyzing change by graphs alone is open to major sources of error. Discusses these sources and explores the issues of serial dependency, unreliability of graphs, interjudge disagreement on graphed data, and the use of time-series statistics with relevance to the counselor in…

Educational software makers are often rebuffed by educational authorities, whose endorsements could lead to governmental stamps of approval, and thus explosive sales. But they usually get warmer receptions in the offices of the nation's school superintendents, who are, after all, their primary customers. The system was not supposed to work this…

School vouchers, particularly since the landmark U.S. Supreme Court ruling on the Cleveland program, are one of the most contentious issues in American education. Seemingly contradictory results across available studies have caused confusion among diverse audiences. The authors suggest that these divergent findings are, in part, due to three…

A central dilemma of portfolio assessment is that as the richness of the data available to readers increases, so do the challenges involved in ensuring acceptable reliability among readers. Drawing on empirical and theoretical work in discourse analysis, ethnomethodology, and other fields, we argue that this dilemma results, in part, from the fact…

The Performance Review, Analysis, and Improvement System for Educators (PRAISE) is a formative evaluation instrument designed to improve the performance of school principals. The system appears to be reliable and valid and is flexible enough to accommodate the needs of a variety of schools. Sample items and categories of the instrument include…

This third of three volumes reports on analytic procedures conducted to address various aspects of the scalar properties of the Device Effectiveness Forecasting Technique (DEFT). DEFT, a series of microcomputer programs applied to data gathered from rating scales, is used to evaluate simulator devices used in U.S. Army weapons training. The…

Since educational statistics, which are relatively easy to obtain, can only attempt to measure "quality," this paper asks how quality in higher education is assessed and how educational rankings, which are defined as benchmarks or attempts to measure, contribute to this process. The paper notes that while attempts to rank institutions of…