With the high penetration of a large number of bidirectional VSCs, power systems gradually have bidirectional power flow and AC/DC coupling characteristics. However, there are fewer studies on dynamic modeling and stability analysis of VSCs considering AC/DC coupling. In this paper, we developed the hybrid parameter matrix model for VSC to describe the AC/DC coupling dynamics. Then, the Nyquist locus-based stability criterion is proposed to study the small-signal stability of a hybrid AC/DC power system. The comparison study shows that the proposed method has a wider range of applications compared to the 3-dimensional impedance analysis, in addition to considering the AC/DC coupling effects. Finally, MATLAB/Simulink simulations verify the accuracy of the proposed method.