As the main cause of irreversible visual impairment, angle-closure glaucoma (ACG) can be detected with anterior segment optical coherence tomography (AS-OCT). The automatic classification of anterior chamber angles (ACA) into closed angle, narrowed angle and open angle in the AS-OCT images is highly significant for understanding glaucoma progression. The traditional techniques for image classification usually rely on the extraction of handcrafted features from the images. Despite the popularity of deep learning methods in image classification, very few researches have been done to utilize them for the multi-class classification of AS-OCT images. In this work, a deep learning based algorithm is proposed for accurate ACA classification in AS-OCT images. The proposed method learns the distinguishable representations from numerous training AS-OCT images using the pyramidal convolution which contains multi-scale of kernels, where each scale includes several kinds of filters with changing depth and size. In this way, it can facilitate capturing the different levels of subtle visual cues that cannot be modeled by the handcrafted features. In addition, the skip connection has been adopted to concatenate the feature maps output at different levels to explore the correlation among them better. Moreover, the hybrid attention module including spatial attention and channel attention is introduced to emphasize important spatial and channel-wise information and reduce redundant information to increase the classification accuracy. We have evaluated classification performance of the proposed algorithm on 2636 ACA images from the Zhongshan Ophthalmic Center, Sun Yat-sen University in China, which is the only public ACG dataset in the world. The experimental results show that our algorithm can classify ACA as closed angle, narrowed angle and open angle effectively and it outperforms existing popular deep learning methods by providing higher specificity, sensitivity, accuracy and balanced accuracy.