Imagine that hundreds of video streams, taken by mobile phones during a rock concert, are uploaded to a server. One attractive application of such prominent dataset is to allow a user to create his own video with a deliberately chosen but virtual camera trajectory. In this paper we present algorithms for the main sub-tasks (spatial calibration, image interpolation) related to this problem. Calibration: Spatial calibration of individual video streams is one of the most basic tasks related to creating such a video. At its core, this requires to estimate the pairwise relative geometry of images taken by different cameras. It is also known as the relative pose problem [1], and is fundamental to many computer vision algorithms. In practice, efficiency and robustness are of highest relevance for big data applications such as the ones addressed in the EU-FET_SME project SceneNet. In this paper, we present an improved algorithm that exploits additional data from inertial sensors, such as accelerometer, magnetometer or gyroscopes, which by now are available in most mobile phones. Experimental results on synthetic and real data demonstrate the accuracy and efficiency of our algorithm. Interpolation: Given the calibrated cameras, we present a second algorithm that generates novel synthetic images along a predefined specific camera trajectory. Each frame is produced from two “neighboring” video streams that are selected from the data base. The interpolation algorithm is then based on the point cloud reconstructed in the spatial calibration phase and iteratively projects triangular patches from the existing images into the new view. We present convincing images synthesized with the proposed algorithm.
KeywordsSpatial CalibrationStructure from MotionVirtual CameraBig Data
Hartley, R. and Zisserman, A. (2003) Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge.
Snavely, N., Seitz, S.M. and Szeliski, R. (2006) Photo Tourism: Exploring Photo Collections in 3D. ACM Transactions on Graphics, 25, 835-846. http://dx.doi.org/10.1145/1141911.1141964
Konolige, K., Agrawal, M. and Sol, J. (2011) Large-Scale Visual Odometry for Rough Terrain. Robotics Research, 66, 201-212.
Giannarou, S., Zhang, Z.Q. and Yang, G.-Z. (2012) Deformable Structure from Motion by Fusing Visual and Inertial Measurement Data. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 7-12 October 2012, 4816-4821. http://dx.doi.org/10.1109/iros.2012.6385671
Fischler, M.A. and Bolles, R.C. (1981) Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Communications of the ACM, 24, 381-395. http://dx.doi.org/10.1145/358669.358692
Gomes, J. (1999) Warping and Morphing of Graphical Objects, Volume 1. Morgan Kaufmann, Burlington.
Lowe, D. (2003) Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 20, 91-110.
Hartley, R.I. (1997) In Defense of the Eight-Point Algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19, 580-593. http://dx.doi.org/10.1109/34.601246
Nister, D. (2004) An Efficient Solution to the Five-Point Relative Pose Problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26, 756-770. http://dx.doi.org/10.1109/TPAMI.2004.17
Hartley, R.I. (1994) Projective Reconstruction and Invariants from Multiple Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 1036-1041. http://dx.doi.org/10.1109/34.329005
Pizarro, O., Eustice, R. and Singh, H. (2003) Relative Pose Estimation for Instrumented, Calibrated Imaging Platforms. Proceedings of the 7th Digital Imaging Computing, Technologies and Applications Conference, Sydney, 10-12 December 2003, 601-612.
Armangu, X. and Salvi, J. (2003) Overall View Regarding Fundamental Matrix Estimation. Image and Vision Computing, 21, 205-220. http://dx.doi.org/10.1016/S0262-8856(02)00154-3
Hartley, R.I. and Kahl, F. (2007) Global Optimization through Searching Rotation Space and Optimal Estimation of the Essential Matrix. Proceedings of the IEEE 11th International Conference on Computer Vision, Rio de Janeiro, 14-21 October 2007, 1-8.
Triggs, B., McLauchlan, P.F., Hartley, R.I. and Fitzgibbon, A.W. (2000) Chapter 21: Bundle Adjustment—A Modern Synthesis. In: Triggs, B., Zisserman, A. and Szeliski, R., Eds., Vision Algorithms: Theory and Practice, Volume 1883, Springer, Berlin, 298-372.
Meer, P. (2004) Robust Techniques for Computer Vision. In: Medioni, G. and Kang, S.B., Eds., Emerging Topics in Computer Vision, Prentice Hall, Upper Saddle River, 107-190.
Rousseeuw, P.J. and Leroy, A.M. (2005) Robust Regression and Outlier Detection. Volume 589. John Wiley & Sons, New York.
Chum, O., Matas, J. and Kittler, J. (2003) Locally Optimized RANSAC. Proceedings of the 25th DAGM Symposium, Magdeburg, 10-12 September 2003, 236-243. http://dx.doi.org/10.1007/978-3-540-45243-0_31
Chum, O. and Matas, J. (2005) Matching with PROSAC—Progressive Sample Consensus. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, 20-25 June 2005, 220-226. http://dx.doi.org/10.1109/cvpr.2005.221
Goshen, L. and Shimshoni, I. (2008) Balanced Exploration and Exploitation Model Search for Efficient Epipolar Geometry Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30, 1230-1242.
Brahmachari, A.S. and Sarkar, S. (2009) Blogs: Balanced Local and Global Search for Nondegenerate Two View Epipolar Geometry. Proceedings of the IEEE 12th International Conference on Computer Vision, Kyoto, 29 September-2 October 2009, 1685-1692.
Raguram, R., Chum, O., Pollefeys, M., Matas, J. and Frahm, J. (2013) USAC: A Universal Framework for Random Sample Consensus. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 2022-2038.
Kneip, M.C.L. and Siegwart, R. (2011) Robust Real-Time Visual Odometry with a Single Camera and an IMU. Proceedings of the British Machine Vision Conference, Dundee, 29 August-2 September 2011, 16.1-16.11. http://dx.doi.org/10.5244/C.25.16
Fraundorfer, F., Tanskanen, P. and Pollefeys, M. (2010) A Minimal Case Solution to the Calibrated Relative Pose Problem for the Case of Two Known Orientation Angles. Proceedings of the 11th European Conference on Computer Vision, Heraklion, 5-11 September 2010, 269-282. http://dx.doi.org/10.1007/978-3-642-15561-1_20
Oskiper, T., Zhu, Z.W., Samarasekera, S. and Kumar, R. (2007) Visual Odometry System Using Multiple Stereo Cameras and Inertial Measurement Unit. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, 17-22 June 2007, 1-8. http://dx.doi.org/10.1109/cvpr.2007.383087
Dalalyan, A. and Keriven, R. (2012) Robust Estimation for an Inverse Problem Arising in Multiview Geometry. Journal of Mathematical Imaging and Vision, 43, 10-23. http://dx.doi.org/10.1007/s10851-011-0281-3
Wolberg, G. (1998) Image Morphing: A Survey. The Visual Computer, 14, 360-372. http://dx.doi.org/10.1007/s003710050148
Gurdan, T., Oswald, M.R., Gurdan, D. and Cremers, D. (2014) Spatial and Temporal Interpolation of Multi-View Image Sequences. Proceedings of the German Conference on Pattern Recognition (GCPR), Münster, 2-5 September 2014, 305-316.
Liu, F., Gleicher, M., Jin, H.L. and Agarwala, A. (2009) Content-Preserving Warps for 3D Video Stabilization. ACM Transactions on Graphics (TOG), 28, 44.
Kopf, J., Cohen, M.F. and Szeliski, R. (2014) First-Person Hyper-Lapse Videos. ACM Transactions on Graphics (TOG), 33, 78. http://dx.doi.org/10.1145/2601097.2601195
Egozi, A., Maass, P. and Sagiv, C. (2015) A Robust Estimation Method for Camera Calibration with Known Rotation. Proceedings of Applied Mathematics and Mechanics (PAMM), 15, to be Published.
Hartley, R. (1993) Extraction of Focal Lengths from the Fundamental Matrix. Unpublished Manuscript.
Kendall, D.G. (1984) Shape Manifolds, Procrustean Metrics, and Complex Projective Spaces. Bulletin of the London Mathematical Society, 16, 81-121. http://dx.doi.org/10.1112/blms/16.2.81
Efros, A.A. and Freeman, W.T. (2001) Image Quilting for Texture Synthesis and Transfer. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, 12-17 August 2001, 341-346.
Stewenius, H., Engels, C. and Nister, D. (2006) Recent Developments on Direct Relative Orientation. ISPRS Journal of Photogrammetry and Remote Sensing, 60, 284-294. http://dx.doi.org/10.1016/j.isprsjprs.2006.03.005
Strecha, C., von Hansen, W., Van Gool, L., Fua, P. and Thoennessen, U. (2008) On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, 23-28 June 2008, 1-8.