Exposure to stress throughout life can cumulatively influence later health, even among young adults. The negative effects of high cumulative stress exposure are well-known, and a shift from episodic to stimulus-response memory has been proposed to underlie forms of psychopathology that are related to high lifetime stress. At the other extreme, effects of very low stress exposure are mixed, with some studies reporting that low stress leads to better outcomes, while others demonstrate that low stress is associated with diminished resilience and negative outcomes. However, the influence of very low lifetime stress exposure on episodic and stimulus-response memory is unknown. Here we use a lifetime stress assessment system (STRAIN) to assess cumulative lifetime stress exposure and measure memory performance in young adults reporting very low and moderate levels of lifetime stress exposure. Relative to moderate levels of stress, very low levels of lifetime stress were associated with reduced use and retention (24 h later) of stimulus-response (SR) associations, and a higher likelihood of using context memory. Further, computational modeling revealed that participants with low levels of stress exhibited worse expression of memory for SR associations than those with moderate stress. These results demonstrate that very low levels of stress exposure can have negative effects on cognition.