The current study investigated the effects of spermatozoa from different sources on pregnancies and neonatal outcomes in intracytoplasmic sperm injection (ICSI) cycles via a retrospective cohort study. A total of 2,512 ICSI cycles, from 2007 to 2015, were classified into the testicular sperm aspiration group (TESA group, n = 148), percutaneous epididymal sperm aspiration group (PESA group, n = 1,031), and ejaculated sperm group (control group, n = 1,333). Higher abnormal fertilization rates, decreased implantation rates, and elevated early miscarriage rates were observed in the PESA group, compared with the control group. Additionally, lower embryo utilization rates in the PESA group were found, compared with the TESA group and control group. Non-ejaculated sperm produced lower good-quality embryo rates than ejaculated sperm. There were no significant differences in clinical pregnancy rates (CPR), live birth rates (LBR), late miscarriage rates, induced labor rates, cumulative CPR, cumulative LBR, neonatal outcomes, and major congenital birth defect risks between the three groups. The current study revealed some less than satisfactory results caused by PESA-ICSI, compared to ejaculated sperm-ICSI or even TESA-ICSI, possibly due to the retrieval of distal senescent sperm by blind aspiration. Hence, microsurgical epididymal sperm aspiration may be reconsidered as an adequate alternative, retrieving high quality motile sperm from the proximal-most epididymal site.