Statistical significance is often misinterpreted to mean replicability or generalizability of results, although a statistically significant difference does not equal a reliable difference. Sample splitting procedures may be a more accurate way of estimating research result generalizability. This type of cross-validation involves randomly dividing the original group and using one subgroup to develop the predictor equation. This equation is used to predict a criterion score for each person in the second subgroup, and predicted criterion scores for subjects in the second subgroup are correlated with actual criterion scores. To illustrate the sample splitting invariance procedure, a small data set (n=42 inservice teachers) from a study by L. G. Daniel and K. R. Okeafor (1987) is used. Results demonstrate that statistical significance testing may not always yield an accurate estimate of replicability when compared to analyses run using the sample splitting invariance procedure. There are five tables of analysis results and an appendix listing relevant computer codes for the computations. A 24-item list of references is included. (SLD)