A full automatic segmentation of video object and tracking scheme based on edge features is proposed in this paper. First of all, change detection and Canny edge detection are applied to get accurate difference edge mask of the inter frame. On the basis of difference edge mask and current edge mask, a binary edge model of the video object is initialized. It is then tracked and updated throughout the sequence to accommodate rotation and changes in shape of the tracked object. At last, these binary models are used to guide the actual video object plane (VOP) extraction by a two-step filling technique. The algorithm is robust to the entire motion and local deformation of object. Experiment results and object evaluation demonstrate that the proposed algorithm is computationally efficient and accurate.