Purpose Histopathologic prognostic factors of rectal cancer are closely associated with local recurrence and distant metastasis. We aim to investigate the feasibility of T2*WI in assessment of clinical prognostic factors of rectal cancer, and compare with DKI. Methods This retrospective study enrolled 50 out of 205 patients with rectal cancer according to the inclusion criteria. The following parameters were obtained: R2* from T2*WI, mean diffusivity (MDk), mean kurtosis (MK), and mean diffusivity (MDt) from DKI using tensor method. Above parameters were compared by Mann-Whitney U-test or students' t test. Spearman correlations between different parameters and histopathological prognostic factors were determined. The diagnostic performances of R2* and DKI-derived parameters were analyzed by receiver operating characteristic curves (ROC), separately and jointly. Results There were positive correlations between R2* and multiple prognostic factors of rectal cancer such as T category, N category, tumor grade, CEA level, and LVI (P < 0.004). MDk and MDt showed negative correlations with almost all the histopathological prognostic factors except CRM and TIL involvement (P < 0.003). MK correlated positively with the prognostic factors except CA19-9 level and CRM involvement (P < 0.006). The AUC ranges were 0.724-0.950 for R2* and 0.755-0.913 for DKI-derived parameters for differentiation of prognostic factors. However, no significant differences of diagnostic performance were found between T2*WI, DKI, or the combined imaging methods in characterizing rectal cancer. Conclusion R2* and DKI-derived parameters were associated with different histopathological prognostic factors, and might act as noninvasive biomarkers for histopathological characterization of rectal cancer.