Childhood adversity may act as a stressor to produce a cascade of neurobiological effects that irreversibly alter neural development, setting the stage for developing psychopathology in adulthood. The catechol-O-methyltransferase (COMT) Val158Met polymorphism has received much attention as a candidate gene associated with environmental adversity, modifying risk for psychopathology. In this study, we aim to see how gene x brain x environment models give a more integrative understanding of brain modifications that contribute to predicting psychopathology related to childhood adversity. A large nonclinical sample of young adults completed Childhood Trauma Questionnaire (CTQ), behavioral scores, multimodal magnetic resonance imaging (MRI) scans, and genotyping. We utilized graph-based connectivity analysis in morphometric similarity mapping and resting-state functional MRI to investigate brain alterations. Relationships among COMT genotypes, CTQ score, imaging phenotypes, and behavioral scores were identified by multiple regression and mediation effect analysis. Significant main effect of CTQ score was found in anatomic connectivity of orbitofrontal cortex that was an outstanding mediator supporting the relationship between CTQ score and anxiety/harm-avoiding personality. We also noted the main effect of childhood trauma on reorganization of functional connectivity within the language network. Additionally, we found genotype x CTQ score interactions on functional connectivity of the right frontoparietal network as well as anatomic connectivity of motor and limbic regions. Our data demonstrate childhood adversity and COMT genotypes are associated with abnormal brain connectivity, structurally and functionally. Early identification of individuals at risk, assessment of brain abnormality, and cognitive interventions may help to prevent or limit negative outcomes.