Early prediction of treatment response to neoadjuvant chemotherapy (NACT) in patients with human epidermal growth factor receptor 2 (HER2)-positive breast cancer can facilitate timely adjustment of treatment regimens. We aimed to develop and validate a Siamese multi-task network (SMTN) for predicting pathological complete response (pCR) based on longitudinal ultrasound images at the early stage of NACT.In this multicentre, retrospective cohort study, a total of 393 patients with biopsy-proven HER2-positive breast cancer were retrospectively enrolled from three hospitals in china between December 16, 2013 and March 05, 2021, and allocated into a training cohort and two external validation cohorts. Patients receiving full cycles of NACT and with surgical pathological results available were eligible for inclusion. The key exclusion criteria were missing ultrasound images and/or clinicopathological characteristics. The proposed SMTN consists of two subnetworks that could be joined at multiple layers, which allowed for the integration of multi-scale features and extraction of dynamic information from longitudinal ultrasound images before and after the first /second cycles of NACT. We constructed the clinical model as a baseline using multivariable logistic regression analysis. Then the performance of SMTN was evaluated and compared with the clinical model.The training cohort, comprising 215 patients, were selected from Yunnan Cancer Hospital. The two independent external validation cohorts, comprising 95 and 83 patients, were selected from Guangdong Provincial People's Hospital, and Shanxi Cancer Hospital, respectively. The SMTN yielded an area under the receiver operating characteristic curve (AUC) values of 0.986 (95% CI: 0.977-0.995), 0.902 (95%CI: 0.856-0.948), and 0.957 (95%CI: 0.924-0.990) in the training cohort and two external validation cohorts, respectively, which were significantly higher than that those of the clinical model (AUC: 0.524-0.588, P all < 0.05). The AUCs values of the SMTN within the anti-HER2 therapy subgroups were 0.833-0.972 in the two external validation cohorts. Moreover, 272 of 279 (97.5%) non-pCR patients (159 of 160 (99.4%), 53 of 54 (98.1%), and 60 of 65 (92.3%) in the training and two external validation cohorts, respectively) were successfully identified by the SMTN, suggesting that they could benefit from regime adjustment at the early-stage of NACT.The SMTN was able to predict pCR in the early-stage of NACT for HER2-positive breast cancer patients, which could guide clinicians in adjusting treatment regimes.Key-Area Research and Development Program of Guangdong Province (No.2021B0101420006); National Natural Science Foundation of China (No.82071892, 82171920); Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application (No.2022B1212010011); the National Science Foundation for Young Scientists of China (No.82102019, 82001986); Project Funded by China Postdoctoral Science Foundation (No.2020M682643); the Outstanding Youth Science Foundation of Yunnan Basic Research Project (202101AW070001); Scientific research fund project of Department of Education of Yunnan Province(2022J0249). Science and technology Projects in Guangzhou (202201020001;202201010513); High-level Hospital Construction Project (DFJH201805, DFJHBF202105).© 2022 The Author(s).