The assessment of cognitive abilities, whether it is for purposes of basic research or applied decision making, is potentially susceptible to both facilitating and debilitating influences. However, relatively little research has examined the degree to which these factors might moderate the criterion-related validity of cognitive ability tests. To address this gap, we use Classical Test Theory formulas to articulate how test anxiety and test familiarity can influence observed scores, observed score variance, and most importantly, the criterion-related validity of observed scores. The resulting equations reveal that understanding the influence of test anxiety and test familiarity on criterion-related validity coefficients requires the consideration of a number of additional parameters. To elucidate the implications of the model, we present a Monte Carlo simulation. Results show that anxiety and familiarity can have a significant negative effect on the observed criterion-related validity, but also show that this effect is highly variable. In particular, the effect depends heavily upon the relation between these factors and the criterion variable. Additionally, we note that the equations we develop highlight important gaps in the literature; there are few clear empirical estimates of several of the parameters in our formulas. We call for future research to better examine these additional relations. (Contains 1 table.)