Introduction: The use of averaged topic-level scores can result in the loss of valuable data and can cause misinterpretation of the effectiveness of system performance. This study aims to use the scores of each document to evaluate document retrieval systems in a pairwise system evaluation. Method: The chosen evaluation metrics are document-level…

This study aims at finding out the effect of peer assessment on the evaluation process of students. The hypothesis underlying this study is that assessment is an integral part of the learning process, which should play an important role in the educational model. The current study will emphasize the importance of using peer assessment as a tool to…

A multiple testing method for examining factorial invariance for latent constructs evaluated by multiple indicators in distinct populations is outlined. The procedure is based on the false discovery rate concept and multiple individual restriction tests and resolves general limitations of a popular factorial invariance testing approach. The…

We evaluate feedback methods for oral presentations used in training non-quantitative research skills (literature review and various associated tasks). Training is provided through a credit-bearing module taught to MSc students of banking, economics and finance in the UK. Monitoring oral presentations and providing "best practice"…

Research aim was to reveal peculiarities of the education of social skills among senior high school age students in physical education classes. We hypothesized that after the end of the educational experiment the senior high school age students will have more developed social skills in physical education classes. Participants in the study were 51…

The motivation for this paper is the authors' recent work on several randomized control trials in which they found the primary result, which averaged across subgroups or sites, to be moderated by demographic or site characteristics. They are led to examine a distinction that the Institute of Education Sciences (IES) makes between "confirmatory"…

In his article, "An Alternative to Null-Hypothesis Significance Tests," Killeen (2005) urged the discipline to abandon the practice of "p[subscript obs]"-based null hypothesis testing and to quantify the signal-to-noise characteristics of experimental outcomes with replication probabilities. He described the coefficient that he…

The authors tested whether students can be taught to better understand conventional representations in diagrams, photographs, and other visual representations in science textbooks. The authors developed a teacher-delivered, workbook-and-discussion-based classroom instructional method called Conventions of Diagrams (COD). The authors trained 1…

The sense that replicability is an important aspect of empirical science led Killeen (2005a) to define "p[subscript rep]," the probability that a replication will result in an outcome in the same direction as that found in a current experiment. Since then, several authors have praised and criticized 'p[subscript rep]," culminating…

This comment offers three descriptions of "p[subscript rep]" that start with a frequentist account of confidence intervals, draw on R. A. Fisher's fiducial argument, and do not make Bayesian assumptions. Links are described among "p[subscript rep]," "p" values, and the probability a confidence interval will capture…

In this paper, I trace the development of statistical significance testing standards in sociology by analyzing data from articles published in two prestigious sociology journals between 1935 and 2000. I focus on the role of two key elements in the diffusion literature, contagion and rationality, as well as the role of institutional factors. I find…

The normative implications of applying significance tests in evaluation research are examined. The authors conclude that evaluators often make normative decisions, based on the traditional .05 significance level in studies with small samples. Additional reporting of the magnitude of impact, the significance level, and the power of the test is…

Many research-related classes in social sciences present probability as a unified approach based upon mathematical axioms, but neglect the diversity of various probability theories and their associated philosophical assumptions. Although currently the dominant statistical and probabilistic approach is the Fisherian tradition, the use of Fisherian…

The assumption of equal treatment implementation is questioned. Through the reanalysis of data from a nutrition supplementation program evaluation, the power of the analysis of treatment effects is shown to increase when data on the level of treatment implementation is included. (Author/CM)

A novel solution to the statistical problems in an evaluation of rare events is described. The significance of variations in the number of child homicides is analyzed in a binary time series of "active" months for monitoring future incidence and related systemic events. (Author/CM)