While it is imperative that attempts be made to assess the predictive accuracy of any prediction model, traditional measures of predictive accuracy have been criticized as suffering from "the base rate problem." The base rate refers to the relative frequency of occurrence of the event being studied in the population of interest, and the problem stems from the fact that statistical prediction models often are not valid when applied to populations with a different base rate than the population for which the prediction model was constructed. This study tested alternative predictive accuracy models, two of which account for base rate levels, to determine the degree to which they are base rate invariant. The indices were the three indices recommended by S. Menard (1995), the Relative Improvement over Change (RIOC) method, and the percentage correct classification indices. A Monte Carlo simulation study was undertaken to generate two types of logistic regression models, one with a dichotomous predictor and a continuously measured predictor and the other with two dichotomous predictors. Four reliabilities, three base rate conditions, and two sample sizes were used. All three of Menard's indices were found to be sensitive to fluctuations in the base rate. Conditions under which these indices and the RIOC may be used are summarized in a table. It is recommended that researchers compute all three of Menard's indices and then compare the values across the three to get an indication of the underlying base rate of the sample. (Contains 3 tables and 26 references.) (SLD)