Breast cancer (BC) is an inflammatory tumor caused by a variety of pathological factors, and is still the most common malignant tumor in women. Immune-related genes (IRGs) play a prominent role in the oncogenesis and progression of BC, and are of tumor-specific expression patterns that would benefit the prognosis evaluation. However, there were no systematic studies concerning the possibilities of IRGs in BC prognosis. In this study, the Cancer Genome Atlas (TCGA) database was used to integrate the expression profiles of IRG with the overall survival (OS) rate of 1039 breast cancer patients. The Cox regression analysis was used to predict the survival-related IRGs in BC. Then, we successfully screened a total of 6 IRGs, including PSME2, ULBP2, IGHE, SCG2, SDC1, and SSTR1, and accordingly constructed a prognosis prediction model of BC. Based on the IRG-related model, the BC patients were divided into high- and low-risk groups, and the association between the prognostic model and tumor immune microenvironment (TME) was further explored. The prognostic model reflected the infiltration of various immune cells. Moreover, the low-risk group was found to be with higher immunophenoscore and distinct mutation signatures compared with the high-risk group. The histological validation showed that SDC1, as well as M2 macrophage biomarker CD206, were both of higher abundance in BC samples of high-risk patients, compared with those of low-risk patients. Our results identify the clinically significant IRGs and demonstrate the importance of the IRG-based immune prognostic model in BC monitoring, prognosis prediction, and therapy.