It is common practice for assessment programs to organize qualifying sessions during which the raters (often known as "markers" or "judges") demonstrate their consistency before operational rating commences. Because of the high-stakes nature of many rating activities, the research community tends to continuously explore new…

The present article considers a fundamental question in evaluation research: "By how much do program effects vary across sites?" The article first presents a theoretical model of cross-site impact variation and a related estimation model with a random treatment coefficient and fixed site-specific intercepts. This approach eliminates…

The relevance of the investigated problem is determined by the need to improving the evaluation procedures in education and the student assessment in the age of the context of education widening, new modes of study developing (such as blending learning, e-learning, massive open online courses), immediate feedback necessity, reliable and valid…

Propensity score analysis (PSA) is a methodological technique which may correct for selection bias in a quasi-experiment by modeling the selection process using observed covariates. Because logistic regression is well understood by researchers in a variety of fields and easy to implement in a number of popular software packages, it has…

The aim of this paper is to present a shortened version of the SIS (support intensity scale) obtained by the application of mathematical models and instruments, adopting special algorithms based on the most recent developments in artificial adaptive systems. All the variables of SIS applied to 1,052 subjects with ID (intellectual disabilities)…

Designs for sequential sampling procedures that adapt to cumulative information are discussed. A familiar illustration is the play-the-winner rule in which there are two treatments; after a random start, the same treatment is continued as long as each successive subject registers a success. When a failure occurs, the other treatment is used until…

A technique to estimate variances of weighted estimates in program evaluation is discussed. BRR estimates of variance are constructed from orthogonally weighted subsamples and can provide better estimates from complex samples than can other methods. Use of BRR is illustrated in a government evaluation of a food service program. (SLD)

A fairly large number of different methods exist for evaluating comparisons or null hypothetical contrasts. Because it is difficult to choose among them, this paper develops a 2 by 2 classification of the methods. The first dimension of the classification is decision-based error rate vs. experimentwise error rate. The second dimension is planned…

The basic design for the national evaluation of the Follow Through program is presented, and some of the related issues of analysis are considered. The design, as it now stands, presents many difficulties for analysis. These analysis issues are seen to include the following: (1) What should be the unit of analysis?; (2) How is the effect of a…

Based on seminal work by Lord and Hambleton, Swaminathan, and Rogers, this article is an analytical, graphical, and conceptual reminder that item response theory (IRT) parameter invariance only holds for perfect model fit in multiple populations or across multiple conditions and is thus an ideal state. In practice, one attempts to quantify the…

Bounds for the tails of Dirichlet integrals are established by showing that each integral as a function of the limits is a Schur function. In particular, it is shown how these bounds apply to the simultaneous analysis of variance test and to the multinomial distribution. (Author)

In an analysis of multitrait-multimethod matrices the criteria for discriminant validity are shown to include a "structure" criterion as an invariance of traits structure to methods. The criterion is meant to fit data to an additive model with traits and methods but not interaction terms. The importance of the structure criterion and the…

RESULTS OF FIFTY DIFFERENT RETRIEVAL METHODS AS APPLIED IN THREE EXPERIMENTAL RETRIEVAL SYSTEMS WERE SUBJECTED TO AN ANALYSIS SUGGESTED BY STATISTICAL DECISION THEORY. THE ANALYSIS USES A PREVIOUSLY-PROPOSED MEASURE OF EFFECTIVENESS AND DEMONSTRATES ITS SEVERAL PROPERTIES. SOME OF THESE PROPERTIES ARE--(1) IT ENABLES THE RETRIEVAL SYSTEM TO…

The equating methods and sampling designs used in the empirical studies in this special issue on the use of matched samples for test equating are described. Four requisites for equating are listed, and the invariance of equating functions requisite is identified as the focus of this issue. (SLD)

The concepts, definitions, and notation that have evolved with the development of telephone survey design methodology are discussed and presented as a unified structure. This structure is then applied to some of the more well-known telephone survey designs and alternative designs are developed. The relative merits of the different survey designs…