Infants prefer to listen to happy speech. To assess influences of speech affect on early lexical processing, 7.5- and 10.5-month-old infants were familiarized with one word spoken with happy affect and another with neutral affect and then tested on recognition of these words in fluent passages. Infants heard all passages either with happy affect or with neutral affect. Contrary to initial expectations that positive affect would facilitate word recognition, younger infants recognized familiarized words only when affect "matched" across familiarization and testing. Older infants displayed a more mature pattern of word recognition, recognizing words across variations in affect regardless of the direction of change when the task was somewhat simplified. However, younger infants continued to be limited by affective matching in the simplified task. Early processing advantages thus do not necessarily follow listening preferences. Rather, infants' early lexical representations appear to be dominated by covarying properties of experienced exemplars, whether or not these are ultimately relevant for lexical distinctions.