Obtaining values for latent variables in factor analysis models, also referred to as factor scores, has long been of interest to researchers. However, many treatments of factor analysis do not focus on inference about the latent variables, and even fewer do so from a Bayesian perspective. Researchers may therefore be ill-acquainted with Bayesian thinking on this issue, despite the fact that certain existing procedures may be seen as Bayesian to some extent. The focus of this paper is to provide a conceptual grounding for Bayesian inference for latent variables, articulating not only what Bayesian inference has to say about values for latent variables, but why Bayesian inference is suited for this problem. As to why, it is argued that the notion of exchangeability motivates the form of factor analysis, as well as Bayesian inference for latent variables. The argument is supported by documenting the widespread use of Bayesian inference in analogous settings, including latent variables in other measurement models, multilevel models, and missing data. As to what, this work describes a Bayesian analysis when other parameters are known, as well as partially and fully Bayesian analyses when other parameters are unknown. This facilitates a discussion of various choices researchers have when adopting Bayesian approaches to inference about latent variables.