Hand and wrist skeletal radiographs serve as an important medium for diversified medical and forensic tasks involving bone age assessment. As an alternative to traditional atlas-based bone age identification techniques, deep learning algorithms automatically classify the radiographs into predefined bone age classes, provided that the deep neural networks (DNN) have been well trained with large scale annotated datasets. Most of the current bone age classification DNNs directly explore the existing network models developed for other computer vision representations and understanding applications, such as VGG, Inception, and ResNet. In this work, we present a multi-scale attention-enhanced classifier with a convolutional neural network backbone, specifically designed for bone age prediction and trained to learn a subject's bone age and gender jointly. The proposed classifier is trained with the dataset provided by the RSNA machine learning challenge, and the low-level semantic features are then transferred to a smaller Tongji dataset collected from a hospital in China. As demonstrated by the experiments, the proposed classifier achieves the MADs of 0.41 years over RSNA data and 0.36 years on Tongji data, outperforming other single model state-of-the-art and baseline algorithms for the same test. It illustrates that joint learning of gender information plays a critical role in refining the bone age assessment, while the convolution-based attention mechanism helps retrieve the key features.