The human brain rapidly develops during the first two years following birth. Quantitative susceptibility mapping (QSM) provides information of iron and myelin variations. It is considered to be a valuable tool for studying brain development in early life. In the present work, QSM is performed on neonates, 1-year and 2-year old infants, as well as a group of adults for the purpose of reference. Age-specific templates representing common brain structures are built for each age group. The neonate and infant QSM templates have shown some unique findings compared to conventional T1w and T2w imaging techniques. The contrast between the gray and white matters on the QSM images did not change through brain development from neonate to adult. A linear correlation was found between brain myelination determined in this study and the microscopic myelin degree determined by a previous autopsy study. Also, the magnetic susceptibility values of the cerebral spinal fluid (CSF) exhibit a gradually decreasing trend from birth to 2 years old and to adulthood. The findings suggest that the macromolecular content, myelin, and iron may play the most important contributing factors for the magnetic susceptibility of neonate and infant brain. QSM can be a powerful means to study early brain development and related pathologies that involve alterations in macromolecular content, iron, or brain myelination.