The stability and nonlinear dynamic behavior of drilling shaft system in copper stave deep hole drilling were analyzed. The effects of the fluctuation of the cutting force, the mass eccentricity and the hydrodynamic forces of cutting fluid could be taken into consideration in the model of drilling shaft system. Based on the isoparametric finite element method, the variational form of Reynolds equation in hydrodynamic fluid was used to calculate nonlinear hydrodynamic forces and their Jacobian matrices simultaneously. In the stability analysis, a new shooting method for rapidly determining the periodic orbit of the nonlinear drilling shaft system and its period was presented by rebuilding the traditional shooting method and changing the time scale. Through the combination of theories with experiment, the correctness and effectiveness of the above methods are verified by using the Floquet theory. The results show that the mass eccentricity can inhibit the whirling motion of drilling shaft to some extent. The stability and nonlinear dynamic behavior of drilling shaft system in copper stave deep hole drilling were analyzed. The effects of the fluctuation of cutting force, the mass eccentricity and the hydrodynamic forces of cutting fluid could be taken into consideration in the model of drilling shaft based on the isoparametric finite element method, the variational form of Reynolds equation in hydrodynamic fluid was used to calculate nonlinear hydrodynamic fluid and their Jacobian matrices simultaneously. In the stability analysis, a new shooting method for rapidly determining the periodic orbit of the nonlinear Drilling shaft system and its period was presented by rebuilding the traditional shooting method and changing the time scale. Through the combination of theories with experiment, the correctness and effectiveness of the above law are verified by using the Floquet theory. The results show that the mass eccentricity can inhibit the whirling motion of drilling shaf t to some extent.