Imaginal perspective switches are often considered to be difficult, because they call for additional cognitive transformations of object coordinates (transformation hypothesis). Recent research suggests that problems can also result from conflicts between incompatible sensorimotor and cognitive object location codes during response specification and selection (interference hypothesis). Three experiments tested contrasting predictions of both accounts. Volunteers had to point to unseen object locations after imagined self-rotations and self-translations. Results revealed larger pointing latencies and errors for rotations as compared to translations, and monotic latency and error increases for both tasks as a function of the disparity of object directions between real and imagined perspective. Provision of advance information about the to-be-imagined perspective left both effects unchanged. These results, together with those from a systematic error analysis, deliver clear support for an interference account of imaginal perspective switches in remembered surroundings.