It is commonly assumed that a significant item analysis (F2) provides an assurance that the treatment effect is generalizable to the population of items from which the items were drawn, which in turn implies that the effect is reasonably general across items. The latter implication is shown to be false, and it is argued that a new test of "generality" rather than generalizability is required. Comments from a number of prominent researchers in the field are provided on a separate website. (Contains 4 figures)