The temporal pole (TP), one of the most expanded cortical regions in humans relative to other primates, plays a crucial role in human language processing. It is also one of the most structurally and functionally asymmetric regions. However, whether the functional architecture of the TP is shared by humans and macaques is an open question. We used spectral clustering algorithms to define a cross-species fine-grained TP atlas with different anatomical connectivity patterns. We identified three similar subregions, two ventral and one dorsal, within the TP in both humans and macaques. The parcellation scheme for the TP was validated using functional gradient mapping, anatomical connectivity and resting-state functional connectivity pattern analysis, and functional characterization. Furthermore, in conjunction with the Allen Human Brain Atlas, we revealed the molecular basis for the functional connectivity patterns of each human TP subregion. In addition, we compared the hemispheric asymmetry in mean gray matter volume, anatomical connectivity fingerprints, and whole brain functional connectivity patterns to reveal the evolutionary differences in the TP and found different asymmetric patterns between humans and macaques. In conclusion, our findings reveal that the asymmetry in structure and connectivity may underpin the hemispheric functional specialization of the brain and provide a novel insight into understanding the evolutionary origin of the TP.