Background To compare the compartmentalized diffusion-weighted models, intravoxel incoherent motion (IVIM) and restriction spectrum imaging (RSI), in characterizing breast lesions and normal fibroglandular tissue. Methods This prospective study enrolled 152 patients with 157 histopathologically verified breast lesions (41 benign and 116 malignant). All patients underwent a full-protocol preoperative breast MRI, including a multi-b-value DWI sequence. The diffusion parameters derived from the mono-exponential model (ADC), IVIM model (Dt, Dp, f), and RSI model (C-1, C-2, C-3, C1C2, F-1, F-2, F-3, F1F2) were quantitatively measured and then compared among malignant lesions, benign lesions and normal fibroglandular tissues using Kruskal-Wallis test. The Mann-Whitney U-test was used for the pairwise comparisons. Diagnostic models were built by logistic regression analysis. The ROC analysis was performed using five-fold cross-validation and the mean AUC values were calculated and compared to evaluate the discriminative ability of each parameter or model. Results Almost all quantitative diffusion parameters showed significant differences in distinguishing malignant breast lesions from both benign lesions (other than C-2) and normal fibroglandular tissue (all parameters) (all P < 0.0167). In terms of the comparisons of benign lesions and normal fibroglandular tissues, the parameters derived from IVIM (Dp, f) and RSI (C-1, C-2, C1C2, F-1, F-2, F-3) showed significant differences (all P < 0.005). When using individual parameters, RSI-derived parameters-F-1, C1C2, and C-2 values yielded the highest AUCs for the comparisons of malignant vs. benign, malignant vs. normal tissue and benign vs. normal tissue (AUCs = 0.871, 0.982, and 0.863, respectively). Furthermore, the combined diagnostic model (IVIM + RSI) exhibited the highest diagnostic efficacy for the pairwise discriminations (AUCs = 0.893, 0.991, and 0.928, respectively). Conclusions Quantitative parameters derived from the three-compartment RSI model have great promise as imaging indicators for the differential diagnosis of breast lesions compared with the bi-exponential IVIM model. Additionally, the combined model of IVIM and RSI achieves superior diagnostic performance in characterizing breast lesions.