Background This retrospective study compared the efficacy and side effect profile between postoperative adjuvant radiotherapy and chemoradiotherapy in stage II or stage III thoracic esophageal squamous cell carcinoma (TESCC) patients who underwent curative (R0) esophagectomy. Methods A total of 272 TESCC patients who underwent radical esophagectomy from 2007 to 2016 were included in this retrospective analysis. All cases were pathologically confirmed with stage II or III disease and 148 patients received postoperative chemoradiotherapy (CRT), while the remaining 124 patients received postoperative radiotherapy (RT) alone. Results In CRT and RT groups, the three-year overall survival rates were 51.3 versus 31.5% (P < 0.01) and the median overall survival (OS) was 39 months (95% CI, 31.6 to 46.3 months) and 30 months (95% CI, 21.0 to 38.9 months), respectively (P = 0.213). Three-year disease-free survival rates (DFS) were 30.5% versus 15.9% (P = 0.008), while the median DFS times were 26 months (95% CI, 17.7 to 34.3 months) and 19 months (95% CI, 16.4 to 21.6 months), respectively (P = 0.156). Univariate and multivariate analyses showed AJCC (American Joint Committee on Cancer seventh edition) stage and N stage were independent prognostic factors for overall survival, while the N stage was an independent prognostic factor for disease-free survival. Conclusions Postoperative chemoradiotherapy led to one- and three-year overall survival benefits along with an obvious increase in treatment side effects for stage II to III TESCC patients, with no further improvement in five-year survival. However, the chemoradiotherapy benefits mainly favor stage III,number of resected lymph nodes less than 15, younger (less than 60 years old) and smoking patients.