Recent epidemiological and experimental studies suggest a link between cognitive decline in late adulthood and sports concussions sustained in early adulthood. In order to provide the first in vivo neuroanatomical evidence of this relation, the present study probes the neuroimaging profile of former athletes with concussions in relation to cognition. Former athletes who sustained their last sports concussion >3 decades prior to testing were compared with those with no history of traumatic brain injury. Participants underwent quantitative neuroimaging (optimized voxel-based morphometry [VBM], hippocampal volume, and cortical thickness), proton magnetic resonance spectroscopy ((1)H MRS; medial temporal lobes and prefrontal cortices), and neuropsychological testing, and they were genotyped for APOE polymorphisms. Relative to controls, former athletes with concussions exhibited: 1) Abnormal enlargement of the lateral ventricles, 2) cortical thinning in regions more vulnerable to the aging process, 3) various neurometabolic anomalies found across regions of interest, 4) episodic memory and verbal fluency decline. The cognitive deficits correlated with neuroimaging findings in concussed participants. This study unveiled brain anomalies in otherwise healthy former athletes with concussions and associated those manifestations to the long-term detrimental effects of sports concussion on cognitive function. Findings from this study highlight patterns of decline often associated with abnormal aging.