Objective. Accurate T staging of rectal cancer based on ultrasound images is convenient for doctors to determine the appropriate treatment. To effectively solve the problems of low efficiency and accuracy of traditional methods for T staging diagnosis of rectal cancer, a deep-learning-based Xception-MS diagnostic model is proposed in this paper.Approach. The proposed diagnostic model consists of three steps. First, the model preprocesses rectal cancer images to solve the problem of data imbalance and deficiency of sample size, and reduces the risk of model overfitting. Second, a new Xception-MS network with stronger feature extraction capability, which is a combination of the Xception network and MS module, is proposed. The MS module is a new function module that can more effectively extract multi-scale information from rectal cancer images. In addition, to solve the deficiency of the small sample size of rectal cancer, the proposed network is combined with transfer learning technology. At last, the output layer of the network is modified, in which the global average pooling and a fully connected softmax layer are employed to replace the original ones, and then the rectal cancer 4 classification (T1, T2, T3, T4 staging) is output.Main results. Experiments of rectal cancer T staging are conducted on a dataset of 1078 rectal cancer images in 4 categories collected from the Department of Colorectal Surgery of the Third Affiliated Hospital of Kunming Medical University. The experimental results show that the accuracy, precision, recall andF1 values obtained by the model are 94.66%, 94.70%, 94.65%, and 94.67%, respectively.Significance. The experimental results show that our model is superior to the existing classification models, can effectively and automatically classify ultrasound images of rectal cancer, and can better assist doctors in the diagnosis of rectal cancer.© 2022 Institute of Physics and Engineering in Medicine.