Background: The coronavirus disease 2019 (COVID-19) has spread worldwide with alarming levels of spread and severity. The distribution of angiotensin converting enzyme 2 (ACE2) and transmembrane protease serine 2 (TMPRSS2) from bioinformatics evidence, the autopsy report for COVID-19 and the published study on sperm quality indicated COVID-19 could have a negative impact on male fertility. However, whether the negative impact of COVID-19 on male fertility is persistent remains unknown, which requires long-term follow-up investigation. Methods: Semen samples were collected from 36 male COVID-19 patients with a median recovery time of 177.5 days and 45 control subjects. Then, analysis of sperm quality and alterations of total sperm number with recovery time were performed. Results: There was no significant difference in semen parameters between male recovered patients and control subjects. And the comparisons of semen parameters between first follow-up and second follow-up revealed no significant difference. In addition, we explored the alterations of sperm count with recovery time. It showed that the group with recovery time of >= 120 and <150 days had a significantly lower total sperm number than controls while the other two groups with recovery time of >= 150 days displayed no significance with controls, and total sperm number showed a significant decline after a recovery time of 90 days and an improving trend after a recovery time of about 150 days. Conclusions: The sperm quality of COVID-19 recovered patients improved after a recovery time of nearly half a year, while the total sperm number showed an improvement after a recovery time of about 150 days. COVID-19 patients should pay close attention to the quality of semen, and might be considered to be given medical interventions if necessary within about two months after recovery, in order to improve the fertility of male patients as soon as possible.