The internal meshing spur gear pair is the research object, and the potential energy method is applied to calculate the time-varying meshing stiffness of the internal gear. The internal gear tooth profile is divided into two parts: involute and transition curves, and the gear tooth stiffness is calculated by numerical integration based on accurate tooth profile, which improves the calculation accuracy. Analyzed the influence of different crack size parameters on the stiffness of internal meshing gears. A coupled dynamic model of a planetary system with internal gear crack faults was established, and the influence of cracks on the dynamic response of the planetary system was studied using Zoom-FFT spectrum and cepstrum analysis methods. The simulation results show that as the crack parameter size increases, the mesh stiffness of the internal gear pair gradually weakens, and the periodic vibration impact of the planetary-internal gear pair is also more severe. Cepstrum analysis can easily capture weak fault characteristic frequency information in the system and discover their variation patterns. The research results can provide a theoretical basis for the state monitoring and fault diagnosis of gear systems with crack defects.