Objectives To evaluate the prediction performance of deep convolutional neural network (DCNN) based on ultrasound (US) images for the assessment of breast cancer molecular subtypes. Methods A dataset of 4828 US images from 1275 patients with primary breast cancer were used as the training samples. DCNN models were constructed primarily to predict the four St. Gallen molecular subtypes and secondarily to identify luminal disease from non-luminal disease based on the ground truth from immunohistochemical of whole tumor surgical specimen. US images from two other institutions were retained as independent test sets to validate the system. The models' performance was analyzed using per-class accuracy, positive predictive value (PPV), and Matthews correlation coefficient (MCC). Results The model achieved good performance in identifying the four breast cancer molecular subtypes in the two test sets, with accuracy ranging from 80.07% (95% CI, 76.49-83.23%) to 97.02% (95% CI, 95.22-98.16%) and 87.94% (95% CI, 85.08-90.31%) to 98.83% (95% CI, 97.60-99.43) for the two test cohorts for each sub-category, respectively. In terms of 4-class weighted average MCC, the model achieved 0.59 for test cohort A and 0.79 for test cohort B. Specifically, the DCNN also yielded good diagnostic performance in discriminating luminal disease from non-luminal disease, with a PPV of 93.29% (95% CI, 90.63-95.23%) and 88.21% (95% CI, 85.12-90.73%) for the two test cohorts, respectively. Conclusion Using pretreatment US images of the breast cancer, deep learning model enables the assessment of molecular subtypes with high diagnostic accuracy.