Video processing is one of the most challenging areas in image processing. It deals with identifying an object of interest. Motion detection has been used in many fields either directly or indirectly. In this paper an efficient approach to motion detection in video sequence using color feature extraction operator. Using this approach we improve the background subtraction and detecting the moving object with greater accuracy. In this paper, background modeling is done in order to make the update of background due to light illumination and change in the weather condition. Foreground detection is done before updating the background model. Color feature extraction is done in order to avoid the dynamic background such as moving leaves, rain, snow, rippling water.