Emerging studies documented the association between ambient air pollution exposure and semen quality, but the critical exposure windows have not been comprehensively studied. To identify susceptible windows for associations of exposure to ambient respirable particulate matter (PM10), nitrogen dioxide (NO2), sulfur dioxide (SO2), and ozone (O-3) with sperm concentration, sperm count, total motility, and progressive motility, we recruited 1061 men attending an infertility clinic in Wuhan, China, between 2011 and 2013. We used a distributed lag multivariate linear regression to assess the exposure-lag-response relationship between semen quality and weekly air pollution exposure. The critical exposure windows were during the 6th to 12th sperm development weeks for PM10, 10th to 11th weeks for O-3, and 0 to 5th weeks for SO2. Over the entire 12 weeks of spermatogenesis period, an interquartile range increase (IQR) increase in PM10 was associated with declined sperm concentration [-45.64% (95% CI: -59.97%, -26.18%) percent decrease], declined sperm count [-49.42% (95% CI: -64.42%, -28.09%) percent decrease], reduced total motility [-12.42 (95% CI: 20.47, -4.37)], and reduced progressive motility [-8.81 (95% CI: -16.00, -1.61)], SO2 per IQR increase was associated with reduced sperm concentration [-39.73% (95% CI: -55.96%, -17.51%) percent decrease] and total motility [-8.64 (95% CI: -16.90, -0.38)], but NO2 and O-3 were not associated with any of the four sperm quality parameters. Our findings suggest that exposure to PM10 during spermatidogenesis period, exposure to SO2 during spermatocytogenesis period, and exposure to O-3 during spermiogenesis period were associated with impaired semen quality, which implies air pollutants impair semen quality through varied pathways.