Background Given the long and costly training cycle required for pedicle screw placement, we proposed an immersive mixed reality surgical self-training system (IMR-SS) for pedicle screw placement. IMR-SS combines holographic real-time training instructions with a physical spine phantom to integrate virtual simulation with hands-on training. This study evaluated the effectiveness of IMR-SS in a randomized controlled trial. We aimed to determine whether IMR-SS improves novice learners' pedicle screw placement performance and training satisfaction. Methods The IMR-SS for pedicle screwing consists of three parts: a teaching module, a hands-on module, and an assessment module. The IMR-SS integrates 3D-printed models, real surgical instruments, and immersive mixed reality technology to provide an immersive learning experience, high-fidelity haptic feedback, and real-time instructions. A randomized controlled trial was conducted with 32 undergraduate medical students from two centers. Participants were randomly assigned to either the IMR group (using IMR-SS) or the control group (using a digital textbook). Both groups underwent theoretical and practical training, followed by identical assessments. Results Compared with the control group, the IMR group presented significantly higher completion rates (0.99 +/- 0.02 vs. 0.87 +/- 0.11, p < 0.01) and fewer errors (0.06 +/- 0.25 vs. 2.13 +/- 1.54, p < 0.01). The IMR group showed better performance with significantly smaller screw placement angles (13.88 degrees +/- 6.98 degrees vs. 20.89 degrees +/- 11.59 degrees, p = 0.049). The theoretical assessment revealed no significant difference between the two groups, indicating equivalent baseline knowledge. Compared with the control group, the IMR group had greater training satisfaction and greater confidence in training outcomes. Conclusions The IMR group reported greater satisfaction with and confidence in the outcomes of screw placement training. The IMR-SS is a feasible and effective method for enhancing surgical operation education for novice medical students, providing superior hands-on training experiences and improving practical skills. Future research should focus on long-term learning curve validation and skill transferability and develop more surgery curricula for generalizability validation.