High intensity focused ultrasound (HIFU) is a non-invasive therapy used to induce thermal dose to target tissue for desired medical outcomes. With this technique, malignant tissues can be destroyed using high energy with minimal side effects compared to surgery, that can induce more pain and leave permanent scar to patients. Temperature monitoring is crucial to preserve healthy tissues during thermal therapies. Magnetic resonance (MR) thermometry is often used to monitor the ablation process precisely. However, it is costly and some patients have contraindications. Ultrasound is a cost-effective medical imaging modality and suffers from less restrictions on the operating environment and can be used safely with all patients. In this paper, we propose a method to monitor the temperature with ultrasound using a deep learning approach. The system is designed to collect ultrasound channel data during HIFU therapy and alternates ablation phases and monitoring phases. In the monitoring phase, ultrasound elements in the probe receive ultrasound pulses sent from the 256 HIFU elements sequentially. We use convolutional long short term memory (ConvLSTM) neural network to generate temperature images from the ultrasound channel data. The temperature images are compared with the ones collected from MR thermometry. Mean and max difference of each image are calculated to evaluate the performance of designed neural network. We achieve 0.57 ± 0.33 °C of mean difference and 1.99 ± 1.07 °C of max difference in axial plane. In coronal plane, we achieve 0.33 ± 0.19 °C of mean difference and 1.54 ± 1.04 °C of max difference. The results show the potential use of ultrasound and deep learning to reconstruct temperature images.