Recently, there has been increased interest in the relationship between cerebral small vessel disease (CSVD) and circadian rhythm disruption, particularly sleep disturbance. However, the neural mechanism of sleep disturbance in CSVD patients remains poorly understood. The purpose of this study is to explore the gray matter alterations in CSVD patients with and without sleep disturbance. 59 patients with CSVD and 40 healthy controls (HC) were recruited for the present study. Sleep quality was assessed using the Pittsburgh Sleep Quality Index (PSQI) questionnaire. CSVD patients were categorized into either the good sleepers group (CSVD-GS, n = 23) or the poor sleepers group (CSVD-PS, n = 36) based on PSQI score. Voxel-based morphometry (VBM) analysis was used to assess differences in gray matter volume (GMV) between groups. Multivariate regression analyses were performed to investigate the relationships between sleep quality, GMV, and white matter hyperintensities (WMH). We observed GMV differences between the three groups in the bilateral caudate, right thalamus, bilateral calcarine cortex, left precentral gyrus, right orbitofrontal cortex, left cingulate gyrus, and right sub-gyral temporal lobe. Additionally, the CSVD-PS group exhibited decreased GMV in the bilateral calcarine cortex yet increased GMV in the right caudate compared to the CSVD-GS group. In fully adjusted models, GMV of the right caudate and bilateral calcarine cortex was associated with sleep quality in CSVD patients. The present study revealed structural brain alterations in CSVD patients with sleep disturbance. These findings may provide novel insights into the neural mechanisms of sleep disturbance in CSVD.