Evidence shows that visual working memory (VWM) is strongly served by attentional mechanisms, whereas other evidence shows that VWM representations readily survive when attention is being taken away. To reconcile these findings, we tested the hypothesis that directing attention away makes a memory representation vulnerable to interference from the test pattern, but only temporarily so. When given sufficient time, the robustness of VWM can be restored so that it is protected against test interference. In 5 experiments, participants remembered a single grating for a later memory test. In the crucial conditions, participants also performed a letter change detection task in between, during the delay period. Experiments 1-3 demonstrate and replicate the predicted effect: The intervening task had an adverse effect on memory performance, but only when the test display appeared immediately after the secondary task. At longer delays of 3.5 s, memory performance was on a par with conditions in which there was no intervening task. By varying the similarity of the test pattern to the memorized pattern, Experiments 4 and 5 further showed that performance suffered at early test intervals, unless the test item was dissimilar to the memory item. The results provide positive evidence for test interference, and demonstrates that the susceptibility to interference only occurs temporarily, as memory is restored when attention is allowed to return to the memorandum.