Even though deep learning is fascinated in fields of coronary vessel segmentation in X-ray angiography and achieves prominent progresses, most of those models probably bring high false and missed detections due to indistinct contrast between coronary vessels and background, especially for tiny sub-branches. Image improvement technique is able to better such contrast, while boosting extraneous information, e.g., other tissues with similar intensities and noise. If incorporating features derived from original and enhanced images, the segmentation performance is improved because those images comprise complementary information from different contrasts. Accordingly, inspired from advantages of contrast improvement and encoding-decoding architecture, a dual multi-scale feature aggregation network (named DFA-Net) is introduced for coronary vessel segmentation in digital subtraction angiography (DSA). DFA-Net integrates the contrast improvement using exponent transformation into a semantic segmentation network that individually accepts original and enhanced images as inputs. Through parameter sharing, multi-scale complementary features are aggregated from different contrasts, which strengthens leaning capabilities of networks, and thus achieves an efficient segmentation. Meanwhile, a risk cross-entropy loss is enforced on the segmentation, for availably decreasing false negatives, which is incorporated with Dice loss for joint optimization of the proposed strategy during training. Experimental results demonstrate that DFA-Net can not only work more robustly and effectively for DSA images under diverse conditions, but also achieve better performance, in comparison with state-of-the-art methods. Consequently, DFA-Net has high fidelity and structure similarity to the reference, providing a way for early diagnosis of cardiovascular diseases.