Aiming at the problems of large buffering and poor dynamic performance of traditional sliding mode observer (SMO), a sensorless control method for brushless DC motor based on a novel SMO is proposed in this paper. On one hand, the arctangent function with smooth curve is selected instead of the sign function to decrease the high frequency buffeting. On the other hand, the sliding mode gain is replaced by an adaptive gain according to the change of the observed current error to ensure the fast convergence and stability of the observer. The stability of the designed SMO is analyzed by Lyapunov stability theory. In addition, G(θ) function, which is known as motor speed independent function method, is used to obtain high signal commutation detection accuracy at low speed. The effect of speed is eliminated by the ratio of different line back electromotive forces, and then the zero crossing of the function is used to judge the commutation point of the motor. Finally, the validity and efficiency of the method are verified by simulation.