It has been debated whether working memory (WM) performance is modulated by the valence of the stimuli that are being processed. A recent meta-analysis revealed that at the behavioral level and in individuals without mental health problems, mean-level performance differences in WM tasks with neutral versus affective conditions are small to negligible. We took this finding an important step further by employing a psychometric approach. This is an important refinement of previous work because even in the absence of mean-level differences, differential processing of affective versus nonaffective information may still be occurring. We examined whether at the construct level, 2 latent WM factors could be distinguished in capturing the processing of neutral and affective stimuli, respectively. Applying confirmatory factor analyses (N = 183 university students) to a battery of 18 tasks (3 n-back paradigms crossed with 3 stimulus types and neutral vs. affective valence), the 2 factors correlated perfectly. This result was replicated when neutral stimuli were analyzed together with either positive or negative stimuli. Based on individual differences, the processing of affective versus nonaffective stimuli in WM therefore cannot be distinguished, at least not in a student sample of younger adults.