Recent reports indicate that immune cells in solid cancers have significant predictive and therapeutic value. IgG4 is a subclass of IgG and we recently found that it exerted an inhibitory effect in tumor immunity. We aimed to assess the significance of IgG4 and T cell subtypes in tumor prognosis. We investigated the density, distribution and relationship of five immune markers CD4, CD8, Foxp3, IL-10 and IgG4 with multiple immunostaining method in 118 esophageal squamous cell carcinoma (ESCC) together with clinical data. The relationship among different immune cell types and with clinical data were analyzed with Kaplan-Meier survival analysis and Cox proportional hazards model to identify independent risk factors among immune and clinicopathological parameters. Five-year survival rate of these patients treated with surgery reached 61%. Higher number of CD4+ plus CD8+ T cells predicted better prognosis (p=0.01) in tertiary lymphoid structure (TLS) and could add to the value of TNM staging. Density of the newly identified immune inhibitor IgG4+ B lymphocytes was found positively correlated to that of CD4+ cells (p=0.02) and IL-10+ cells (p=0.0005), but number of infiltrating IgG4+ cells by itself was not an independent factor for prognosis. However, increased serum concentration of IgG4 indicated a poor prognosis of ESCC (p=0.03). 5-year survival rate of esophageal cancer after surgery has been significantly improved. Increased T cells in TLS predicted better survival, suggesting that T cells in TLS may actively participate in anti-tumor immunity. Serum IgG4 could be a useful predictor of prognosis.