Statistical video modeling is a fundamental issue that plays an important role in many content-based video analysis applications. Recent research shows that Gaussian mixture models (GMMs) offer good capability and flexibility of joint spatio-temporal video modeling. However, the major bottleneck of this kind of approaches is the high computational complexity and possible over-fitting during model learning. In this work, we have improved the efficiency and robustness of GMM training by incorporating key-frame extraction for model learning. Specifically, key-frame extraction is formulated as a feature selection process for video segmentation. Hence we have proposed a new video generative model that embeds frame saliency in GMM-based video modeling. The proposed video model is able to speed up the GMM training significantly by optimally selecting a set of video frames with strong saliency. More interestingly, all video frames can be characterized by a saliency indicator that exhibit specific joint spatio-temporal characteristics of visual features. This new video model may also lead to new functionalities for content-based video analysis. 

Related Publications

• X. Song and G. Fan, "Selecting Salient Frames for Spatio-temporal Video Segmentation", IEEE Trans. on Image Processing, Vol. 16, No. 12,  Dec. 2007.

• L. Liu and G. Fan, “Combined Key-frame and Object-based Video Segmentation”, IEEE Trans. Circuits and System for Video Technology, Volume 15, No. 7, pp869 – 884, July, 2005.

• X. Song and G. Fan, "Key-frame Extraction for Object-based Video Segmentation", in Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2005), Philadelphia, PA, March 2005.

• X. Song and G. Fan, “Joint Key-frame Extraction and Object Segmentation for Content-based Video Analysis”, IEEE Trans. on Circuits and Systems for Video Technology, Vol. 16, No. 7, pp904- 914, July 2006.

• X. Song and G. Fan, "Joint Key-frame Extraction and Object-based Video Segmentation", in Proc. of IEEE Workshop on Motion and Video Computing (MOTION 2005), Breckenridge, Colorado, Jan. 5-6, 2005.

• X. Song and G. Fan, “A New Video Analysis Approach for Coherent Key-frame Extraction and Object Segmentation”, in Proc. of IEEE International Workshop on Multimedia Signal Processing, Shanghai, China, Oct-Nov. 2005.