Transient epileptic amnesia (TEA) is a recently described epilepsy syndrome characterized by recurrent episodes of isolated memory loss. It is associated with two unusual forms of interictal memory impairment: accelerated long-term forgetting (ALF) and autobiographical amnesia. We investigated the neural basis of TEA using manual volumetry and automated multi-atlas-based segmentation of whole-brain magnetic resonance imaging data from 40 patients with TEA and 20 healthy controls. Both methods confirmed the presence of subtle, bilateral hippocampal atrophy. Additional atrophy was revealed in perirhinal and orbitofrontal cortices. The volumes of these regions correlated with anterograde memory performance. No structural correlates were found for ALF or autobiographical amnesia. The results support the hypothesis that TEA is a focal medial temporal lobe epilepsy syndrome but reveal additional pathology in connected brain regions. The unusual interictal memory deficits of TEA remain unexplained by structural pathology and may reflect physiological disruption of memory networks by subclinical epileptiform activity.