Inter-rater reliability (IRR), which is a prerequisite of high-quality ratings and assessments, may be affected by contextual variables, such as the rater's or ratee's gender, major, or experience. Identification of such heterogeneity sources in IRR is important for the implementation of policies with the potential to decrease measurement error…

We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error…