Methods of probability modeling to analyze rater agreement are described, emphasizing their basic similarities and viewing them as variants of a common methodology. Statistical techniques for analyzing agreement data are described to address questions such as how many opinions are required to make a medical diagnosis with necessary accuracy. Kappa and other agreement indices, variance components approaches, and latent structure models are considered. Focus is on two related techniques, which differ in assumptions about disease subtypes and associated differences among cases in their ability to be correctly diagnosed: (1) latent class agreement analysis; and (2) latent trait agreement analysis. Specifically, these methods make it possible to determine from the opinion of panels of diagnosticians in an agreement study the following: the probable accuracy of an individual diagnosis; the probability of disease presence or absence given unanimous or conflicting opinions by several diagnosticians; and how many opinions should be required to make the diagnosis. It is concluded that because the estimation procedures and software are better developed for the latent class agreement model, investigators should pursue this approach first. Eleven data tables and a 54-item list of references are included. (RLC)