Indirect questioning attempts to overcome social desirability bias in survey research. However, to properly analyze the resulting data, it is crucial to understand how it impacts responses. This study analyzes results from a randomized experiment that tests whether direct versus indirect questioning methods lead to different results in a sample of 8,426 youths in Kenya and Pakistan. Through an examination of differential item functioning and regression analyses, we find that question wording leads to differences in how scales should be scored. We conclude that the use of indirect questioning should be undertaken with caution as a method to replace direct questioning.