Background Our objective was to characterize the relationship of anemia and hemoglobin concentrations with cross-sectional cognitive functions and changes in cognitive functions over 2 years in a large sample of Chinese middle aged and elderly. Methods Ten thousand nine hundred eighteen adults aged 45 years or older participating in the China Health and Retirement Longitudinal Study (CHARLS) were used for cross-sectional analyses and 9324 were used for longitudinal analysis. Cognitive functions were assessed by memory recall (episodic memory), mental status (TICS), and global cognitive function at baseline survey (Visit 1) and first follow-up survey (Visit 2). The lower the cognitive test score, the worse the cognitive function. Anemia was defined as hemoglobin concentrations lower than 13 g/dl for men and lower than 12 g/dl for women. Adjusted multivariate regression analyses were used to explore the relationships of different cognitive domains with anemia and hemoglobin concentration. Results Overall, the prevalence of anemia was 12.86% and the mean hemoglobin concentration was 14.37 +/- 2.20 g/dl. After adjusting for socio-demographic and health-related covariates, the cross-sectional association between anemia and global cognitive function [beta (95%CI) = - 0.49(- 0.69 similar to - 0.29)], episodic memory [beta (95%CI) = - 0.14(- 0.23 similar to - 0.05)], and TICS [beta (95%CI) = - 0.23(- 0.38 similar to - 0.08)] were significant and did not differ by gender. The hemoglobin concentration was also associated with global cognitive function among the whole sample (P < 0.05 for all). The longitudinal analyses showed global cognitive function and episodic memory were associated with anemia independent of covariates (P < 0.05 for all). Sensitivity analyses further provided significant results showing the association btween anemia and cognition decline (P < 0.05). Conclusion There was a cross-sectional and longitudinal association between anemia and accelerated decline in cognitive functions in Chinese middle-aged and elderly. This suggests that anemia and low hemoglobin concentrations are independent risk factors of cognitive decline.