This study presents results from a randomized experiment in the 2015-2017 National Survey of Family Growth, where a large national sample of U.S. individuals aged 15-49 was randomly assigned to one of two different versions of a survey question about sexual identity (one with three response options, including heterosexual, gay/lesbian, and bisexual, and one adding the option "something else"). Analyses of changes in the associations of sexual identity with alcohol, tobacco, and other drug use across these treatments revealed evidence of significant differences in the associations that remained robust after adjusting for socio-demographics. The results suggest that when individuals choose their sexual identity from a more limited number of response options, the heterogeneity of the sexual identity subgroups increases, weakening estimated associations of sexual identity with these behaviors. Open-ended questions may therefore be necessary to measure sexual identity and estimate its associations with substance use behaviors accurately in surveys.