Abstract—Electrocardiogram (ECG) effectively records the difference between body potentials generated during the physiological function of the heart. Both ECG and heartbeat sounds are viewed as powerful tools to diagnose abnormal arrhythmias. In the past, the accuracy of such diagnoses has been significantly improved due to the development of machine-learning algorithms. However, current models still do not provide acceptable performance due to similarities of signal waveforms as well as ambient noises and interferences. In this paper, we propose a novel deep-learning model that incorporates a WaveNet model based on dilated convolutions as the backbone followed by multiple bi-directional long-short-term memory (Bi-LSTM) layers to further enhance the discriminant capabilities of temporal relations. A typical clinical dataset, i.e., the MIT-BIH arrhythmia database, which considers intra-patient and inter-patient paradigms based on the American Association of Medical Instrumentation (AAMI) EC57 standard, is used to demonstrate the performance of the proposed approach. Numerical results show that our model has achieved the state-of-the-art classification accuracies.