An Evolutionary Approach to Automatic Video Editing
Tinghuai Wang and Andrew Mansfield and Rui Hu and John P. Collomosse: An Evolutionary Approach to Automatic Video Editing. Conference for Visual Media Production. CVMP '09, pp. 127-134, 12-13 November 2009.
Digital video has become affordable and attractive to home users, but skill and manual labour are still required to transform amateur footage into aesthetically pleasing movies. We present a novel algorithm for transforming raw home video footage into concise, temporally salient clips. We interpret the sequence of editing operations applied to footage as a `program' comprising cutting, panning and zooming constructs. We develop a Genetic Programming (GP) framework for representing and evolving such programs. Under this framework, the search for an aesthetically pleasing video edit becomes a search for the optimal genetic program. Our aesthetic criterion promotes the inclusion of people in shots, whilst penalising rapid shot changes or shot changes in the presence of camera motion. We present results on some representative home videos.
J. Koza, "Genetic programming: A paradigm for genetically breeding populations of computer programs to solve problems," in Stanford University Computer Science Department technical report STAN-CS-90-1314, 1990.
D. Goldberg, Genetic Algorithms in Search Optimization and Machine Learning. Addison-Wesley, 1989.
A. Nagasaka and Y. Tanaka, "Automatic video indexing and full-video search for object appearances," in Proc. VDB, 1991, pp. 113-127.
D. DeMenthon, V. Kobla, and D. Doermann, "Video summarization by curve simplification," in ACM Multimedia. New York, NY, USA: ACM, 1998, pp. 211-218. http://dx.doi.org/10.1145/290747.290773
R. Lienhart, "Abstracting home video automatically," in ACM Multimedia. New York, NY, USA: ACM, 1999, pp. 37-40.
A. Girgensohn, J. Boreczky, P. Chiu, J. Doherty, J. Foote, G. Golovchinsky, S. Uchihashi, and L. Wilcox, "A semiautomatic approach to home video editing," in UIST '00: Proceedings of the 13th annual ACM symposium on User interface software and technology. New York, NY, USA: ACM, 2000, pp. 81-89.
A. Girgensohn, S. Bly, F. Shipman, J. Boreczky, and L.Wilcox, "Home video editing made easy balancing automation and user control," in In Human-Computer Interaction INTERACT '01. IOS. Press, 2001, pp. 464-471.
X. Hua, L. Lu, and H. Zhang, "Optimization-based automated home video editing system," IEEE Trans. Circuits Syst. Video Techn., vol. 14, no. 5, pp. 572-583, 2004. http://dx.doi.org/10.1109/TCSVT.2004.826750
Y. Ma, L. Lu, H. Zhang, and M. Li, "A user attention model for video summarization," in ACM Multimedia. New York, NY, USA: ACM, 2002, pp. 533-542.
T. Mei, X. Hua, H. Zhou, and S. Li, "Modeling and mining of users' capture intention for home videos," IEEE Transactions on Multimedia, vol. 9, no. 1, pp. 66-77, 2007. http://dx.doi.org/10.1109/TMM.2006.886357
M. Al-Hames, B. Hornler, R. Muller, J. Schenk, and G. Rigoll, "Automatic multi-modal meeting camera selection for video-conferences and meeting browsers," in Proc. ICME, 2007, pp. 2074-2077.
T. Hospedales and O. Williams, "An adaptive machine director," in Proc. British Machine Vision Conference (BMVC), 2008.
D. Arijon, Grammar of the Film Language. Silman-James Press, 1991.
P. Viola and M. Jones, "Robust real-time face detection," Int. J. Comput. Vision, vol. 57, no. 2, pp. 137-154, 2004. http://dx.doi.org/10.1023/B:VISI.0000013087.49260.fb
V. Ferrari, M. Marin-Jiminez, , and A. Zisserman, "Progressive search space reduction for human pose estimation," in Proc. CVPR. IEEE, June 2008, pp. 1-8.
R. Oami, A. Benitez, S. Chang, and N. Dimitrova, "Understanding and modeling user interests in consumer videos," in Proc. ICME, 2004, pp. 1475-1478.
R. Poli, W. Langdon, and N. McPhee, A Field Guide to Genetic Programming. Lulu, 2008.
J. Koza and R. Poli, Search Methodologies: Introductory Tutorials in Optimization and Decision Support Techniques. Springer, 2005.
Chapter 3 in PhD Thesis: Tinghuai Wang: Computer Vision for the Structured Representation and Stylisation of Visual Media Collections. Department of Electronic Engineering, University of Surrey, UK, July 2012. https://sites.google.com/site/tinghuaiw/thesis http://sdrv.ms/1fgfCWo