People vary in their performance on visual working memory tasks, and these individual differences covary with a wide range of higher-level cognitive processes including fluid intelligence. Performance also varies across study displays, purportedly driven by both low- and higher-level processes. Understanding what causes these sources of systematic variability has been crucial for developing theories of working memory. However, here we find that all such variability in performance on a test of visual working memory can be accounted for by concurrent variability in visual iconic memory: A person with relatively high working memory capacity will have high iconic memory capacity, and a particularly easy working memory display will also be easy under iconic memory conditions. These results are supported by a nonparametric factor analysis and hierarchical Bayesian model comparison. In a second experiment the relationship between iconic and working memory holds even when they are measured with substantially different experimental paradigms, and a third experiment suggests that the relationship between tests of iconic and working memory is driven by mechanisms other than iconic or working memory storage, such as variation in perceptual or attentional processes.