Background. Cellular heterogeneity within the tumor microenvironment is essential to tumorigenesis and tumor development. A high-resolution global view of the tumor-infiltrating immune and stromal cells in breast tumors is needed. Methods. xCell was used to create a cellular heterogeneity map of 64 cell types in 1,092 breast tumor and adjacent normal tissues. xCell digitally dissects tissue cellular heterogeneity based on gene expression. Integrated statistical analyses were then performed. Results. There were noticeable differences between the cell fractions in tumor tissues and normal tissues. Tumors displayed higher proportions of immune cells, including CD4+ Tem, CD8+ naive T cells, and CD8+ Tcm compared with normal tissues. Immune inhibitory receptors (PD1, CTLA4, LAG3 and TIM3) were co-expressed on certain subtypes of T cells in breast tumors, and PD1 and CTLA4 were both positively correlated with CD8+ Tcm and CD8+ T cells. 28 cell types were significantly associated with overall survival in univariate analysis. CD4+ Tem, CD8+ Tcm, CD8+ T-cells, CD8+ naive T-cells, and B cells were positive prognostic factors but CD4+ naive Tcells were negative prognostic factors for breast cancer patients. TDRD6 and TTK are promising T cell and B cell targets for tumor vaccines. Endothelial cells and fibroblasts were significantly less prevalent in tumor tissues; astrocytes and mesangial cells were negatively correlated with the T stage. Mesangial cells and keratinocytes were found to be favorable prognostic factors and myocytes were negative prognostic factors. Five cell types were found to be independent prognostic factors and we used these to create a reliable prognostic model for breast cancer patients. Cellular heterogeneity was discovered among different breast cancer subtypes by Her2, ER, and PR status. Trinegative patients had the highest fraction of immune cells while luminal type patients had the lowest. The various cells may have diverse or opposing roles in the prognosis of breast cancer patients. Conclusions. We created a unique cellular map for the diverse heterogeneity of immune and stromal phenotypes within the breast tumor microenvironment. This map may lead to potential therapeutic targets and biomarkers with prognostic utility.