This paper proposes a deep-learning based workflow for stenosis classification and localization on coronary angiography images of 194 patients from a multi-center study. Coronary stenosis severities were categorized into three groups of <25% stenosis, 25 to 99% stenosis, total occlusion as 3-CAT and two groups of <25% stenosis, >25% stenosis as 2-CAT for classification training labels; and stenosis bounding boxes were annotated in images as stenosis localization labels, based on expert physician's visual reading on right coronary artery (RCA) and left coronary artery (LCA) images with full contrast filling of the coronary artery. CNN and recurrent neural network models were employed for coronary artery view classification, candidate frame selection and stenosis classification. Furthermore, stenosis activation maps were implemented for weakly-supervised stenoses positioning. In experiments, our method achieved 0.91/0.85 AUC values for 3-CAT stenosis classification in RCA and LCA respectively; and 0.91/0.87 AUC values for 2-CAT classification in RCA and LCA respectively. For stenosis detection on most significant regions, sensitivity for RCA/LCA were 0.72/0.60 respectively; and mean square error between detection and ground-truth center points were 69.6/79.5 pixels for RCA/LCA in image with size of 512. The results show our method achieves high performance in stenosis severity classification and performs reasonably well for stenosis positioning.