Literature screening is the process of identifying all relevant records from a pool of candidate paper records in systematic review, meta-analysis, and other research synthesis tasks. This process is time consuming, expensive, and prone to human error. Screening prioritization methods attempt to help reviewers identify most relevant records while…

In secure web application development, the role of web services will not continue if it is not trustworthy. Retaining customers with applications is one of the major challenges if the services are not reliable and trustworthy. This article proposes a trust evaluation and decision model where the authors have defined indirect attribute, trust,…

The subject of the research is to build methodologies to evaluate the student knowledge by testing. The author points to the importance of feedback about the mastering level in the learning process. Testing is considered as a tool. The object of the study is to create the test system models for defence practice problems. Special attention is paid…

Laufer and Nation (1995) proposed that the Lexical Frequency Profile (LFP) can estimate the size of a second-language writer's productive vocabulary. Meara (2005) questioned the sensitivity and the reliability of LFPs for estimating vocabulary sizes, based on the results obtained from probabilistic simulations of LFPs. However, the underlying…

To assess cross-observer agreement on personality profiles, an Index of Profile Agreement and an associated coefficient are proposed that take into account both the difference between the ratings and the extremes of their mean. Data from the Revised NEO Personality Inventory for 250 peer ratings/self-reports and 68 spouse ratings/self-reports…

Methods of probability modeling to analyze rater agreement are described, emphasizing their basic similarities and viewing them as variants of a common methodology. Statistical techniques for analyzing agreement data are described to address questions such as how many opinions are required to make a medical diagnosis with necessary accuracy. Kappa…

In Dyer's Student Change Model, performance indicators (PIs) are utilized as a measure of the effectiveness of educational programs. These PIs are examined and analyzed in some detail in this paper. In particular, the reliability of the residuals, the sensitivity of the measure to sample size, the random half reliability, and the stability of the…

In this paper, academic efficiency is operationally defined and a methodology for measuring it at the school level is described. Academic efficiency is defined as the extent to which a school utilizes its time for the academic development of all its students. The measure of academic efficiency must include three elements: time, students, and…

A questionnaire is designed to allow assessment of a simple additive value function for testing respondent preferences for different types of information used in evaluating police services. Responses are analyzed to determine what types of information different stakeholder groups consider useful. (Author/LMO)

Statistical significance and "abnormality" have been used as criteria for the evaluation of intra-individual subtest score differences. Shortcomings of these criteria are identified, and improved estimates of the true score differences are suggested. The applicability of the abnormality criterion to these improved estimates is reviewed.…

This paper considers, from a theoretical point of view, two measurement approaches used in measuring success and failure in skills tests in physical education. The first, "fixed length" (FL) testing, entails counting the number of successful performances in a fixed number of trials. The second, "trials-to-criterion" (TTC)…

The use of three remedies for errors in the measurement of ability that arise from differences in rater stringency is discussed. Models contrasted are: (1) Conventional; (2) Handicap; and (3) deterministic Rater Response Theory (RRT). General model requirements, power, bias of measures, computing cost, and complexity are contrasted. Contrasts are…

The way in which trait and rater variance combine in multitrait-multirater (MTMR) performance appraisal data is explored. Implications of the confirmatory factor analytic model and the composite direct product (CDP) model for MTMR data are examined. Superior fit of the CDP model for four data sets is discussed. (SLD)

This document contains three papers from the Methodology Project of the Center for the Study of Evaluation. Methods for characterizing test accuracy are reported in the first two papers. "Bounds on the K Out of N Reliability of a Test, and an Exact Test for Hierarchically Related Items" describes and illustrates how an extension of a…

Issues concerning the selection of judges for standard setting are discussed. Determining the consistency of judges' recommendations, or their congruity with other expert recommendations, would help in selection. Enough judges must be chosen to allow estimation of recommendations by an entire population of judges. (SLD)