Adolescent idiopathic scoliosis (AIS) is a three-dimensional spine deformity governed of the spine. A child's Risser stage of skeletal maturity must be carefully considered for AIS evaluation and treatment. However, there are intra-observer and inter-observer inaccuracies in the Risser stage manual assessment. A multi-task learning approach is proposed to address the low precision issue of manual assessment. With our developed multi-task learning approach, the iliac area is extracted and forwarded to the improved Swin Transformer for Risser stage assessment. The spatial and channel reconstruction convolutional Swin block is adapted to each stage of the Swin Transformer to achieve better performance. The Risser stage assessment based on iliac region extraction had an overall accuracy of 81.53%. The accuracy increased in comparison to ResNet50, ResNet101, Uni-former, Next-ViT, ConvNeXt, and the original Swin Transformer by 5.85%, 4.6%, 3.6%, 2.7%, 2.25%, and 1.8%, respectively. The Grad-CAM visualization is used to understand the interpretability of our proposed model. The results show that the proposed multi-task learning strategy performs well on the Risser stage assessment. Our proposed automatic Risser stage assessment method benefits the clinical evaluation of AIS. Project address: https://github.com/xyz911015/Risser-stage-assessment