We describe four fundamental challenges that complex real-life Virtual Reality (VR) productions are facing today (such as multi-camera management, quality control, automatic annotation with cinematography and 360 ˚ depth estimation) and describe an integrated solution, called Hyper 360, to address them. We demonstrate our solution and its evaluation in the context of practical productions and present related results.
KeywordsDeep LearningTensor FlowYoloV3360&#730VideoVirtual RealityFree Viewpoint VideoQuality ControlAutomatic CinematographyMulti-Camera Systems
Sheikh, A., Brown, A., Evans, M. and Watson, Z. (2016) Directing Attention in 360-Degree Video. Proceedings of IBC 2016 Conference, Amsterdam, 8-12 September 2016, 1-9. https://doi.org/10.1049/ibc.2016.0029
Tang, A. and Fakourfar, O. (2017) Watching 360˚ Videos Together. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, ACM, New York, 4501-4506. https://doi.org/10.1145/3025453.3025519
Fonseca, D. and Kraus, M. (2016) A Comparison of Head-Mounted and Hand-Held Displays for 360° Videos with Focus on Attitude and Behaviour Change. In: Proceedings of the 20th International Academic Mindtrek Conference, ACM, New York, 287-296. https://doi.org/10.1145/2994310.2994334
Warren, M. (2017) Making Your First 360 Video? Here Are 10 Important Things to Keep in Mind. https://www.filmindependent.org/blog/making-first-360-video-10-important-things-keep-mind/
Elmezeny, A., Edenhofer, N. and Wimmer, J. (2018) Immersive Storytelling in 360-Degree Videos: An Analysis of Interplay between Narrative and Technical Immersion. Journal of Virtual Worlds, 11, No. 1. https://doi.org/10.4101/jvwr.v11i1.7298
Malik, R. and Bajcsy, P. (2008) Automated Placement of Multiple Stereo Cameras. The 8th Workshop on Omnidirectional Vision, Camera Networks and Non-Classical Cameras—OMNIVIS, October 2008, Marseille, France.
Chikkerur, S., Sundaram, V., Reisslein, M. and Karam, L.J. (2011) Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison. IEEE Transactions on Broadcasting, 57, 165-182. https://doi.org/10.1109/TBC.2011.2104671
Lai, W.S., Huang, Y., Joshi, N., Buehler, C., Yang, M.H. and Kang, S.B. (2018) Semantic-Driven Generation of Hyperlapse from 360 Degree Video. IEEE Transactions on Visualization and Computer Graphics, 24, 2610-2621. https://doi.org/10.1109/TVCG.2017.2750671
Song, S., Zeng, A., Chang, A.X., Savva, M., Savarese, S. and Funkhouser, T. (2018) Im2Pano3D: Extrapolating 360˚ Structure and Semantics beyond the Field of View. 2018 the IEEE Conference on Computer Vision and Pat tern Recognition, Salt Lake City, UT, 18-23 June 2018, 3847-3856. https://doi.org/10.1109/CVPR.2018.00405
Hyper 360 Project (2019) http://www.Hyper360.eu/
Adobe (2019) Creative Cloud Premier Pro & After Effects. https://www.adobe.com/creativecloud/video/virtual-reality.html
Andersson Technologies (2019) SynthEyes 3D Camera Tracking and Stabilization Software. https://www.ssontech.com/synovu.html
Kopf, J. (2016) 360˚ Video Stabilization. ACM Transactions on Graphics, 35, Article No. 195. https://dl.acm.org/citation.cfm?id=2982405 https://doi.org/10.1145/2980179.2982405
Matos, T., Nóbrega, R., Rodrigues, R. and Pinheiro, M. (2018) Dynamic Annotations on an Interactive Web-Based 360 & Deg; Video Player. In: Proceedings of the 23rd International ACM Conference on 3D Web Technology, ACM, New York, Article 22. https://doi.org/10.1145/3208806.3208818
Su, Y.-C., Jayaraman, D. and Grauman, K. (2017) Pano2Vid: Automatic Cinematography for Watching 360˚ Videos. In: Bares, W., Gandhi, V., Galvane, Q. and Ronfard, R., Eds., Eurographics Workshop on Intelligent Cinematography and Editing, The Eurographics Association, Lyon, France.
Su, Y.-C. and Grauman, K. (2017) Making 360˚ Video Watchable in 2D: Learning Videography for Click Free Viewing. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 21-26 July 2017, 1368-1376.
Hu, H.-N., Lin, Y.-C., Liu, M.-Y., Cheng, H.-T., Chang, Y.-J. and Sun, M. (2017) Deep 360 Pilot: Learning a Deep Agent for Piloting through 360˚ Sports Videos. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, ACM, New York, 1396-1405.
Truong, A., Chen, S., Yumer, E., Li, W. and Salesin, D. (2018) Extracting Regular FOV Shots from 360 Event Footage. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM, New York, 316. https://doi.org/10.1145/3173574.3173890
Huang, J., Chen, Z., Ceylan, D. and Jin, H. (2017) 6-DOF VR Videos with a Single 360-Camera. 2017 IEEE Virtual Reality, Los Angeles, CA, 18-22 March 2017, 37-44. https://doi.org/10.1109/VR.2017.7892229
Wang, F.-E., Hu, H.-N., Cheng, H.-T., Lin, J.-T., Yang, S.-T., Shih, M.-L., Chu, H.-K. and Sun, M. (2018) Self-Supervised Learning of Depth and Camera Motion from 360˚ Videos. CoRR, abs/1811.05304.
Thatte, J., Boin, J.B., Lakshman, H. and Girod, B. (2016) Depth Augmented Stereo Panorama for Cinematic Virtual Reality with Head-Motion Parallax. 2016 IEEE International Conference on Multimedia and Expo, Seattle, WA, 11-15 July 2016, 1-6. https://doi.org/10.1109/ICME.2016.7552858
Zioulis, N., Karakottas, A., Zarpalas, D. and Daras, P. (2018) OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas. In: Ferrari, V., Hebert, M., Sminchisescu, C. and Weiss, Y., Eds., Computer Vision—ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, Springer, Cham, 448-465. https://doi.org/10.1007/978-3-030-01231-1_28
Youtube (2017) Hot and Cold: Heatmaps in VR. https://youtube-creators.googleblog.com/2017/06/hot-and-cold-heatmaps-in-vr.html
Tsatsou, D., Dasiopoulou, S., Kompatsiaris, I. and Mezaris, V. (2014) LiFR: A Lightweight Fuzzy DL Reasoner. In: Presutti, V., Blomqvist, E., Troncy, R., Sack, H., Papadakis, I. and Tordai, A., Eds., The Semantic Web: ESWC 2014 Satellite Events. ESWC 2014. Lecture Notes in Computer Science, Springer, Cham, 263-267. https://doi.org/10.1007/978-3-319-11955-7_32
Redmon, J. and Farhadi, A. (2018) YOLOv3: An Incremental Improvement. Computer Science, arXiv: 1804.02767. http://arxiv.org/abs/1804.02767
Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D. and Bischof, H. (2009) Anisotropic Huber-L1 Optical Flow. In: Cavallaro, A., Prince, S. and Alexander, D., Eds., Proceedings of the British Machine Vision Conference, BMVA Press, London, 108.1-108.11. https://doi.org/10.5244/C.23.108
Karakottas, A., Zioulis, N., Zarpalas, D. and Daras, P. (2018) 360D: A Dataset and Baseline for Dense Depth Estimation from 360 Images. 1st Workshop on 360o Perception and Interaction, European Conference on Computer Vision (ECCV), Munich, Germany, 8-14 September 2018, 1-4.
Handa, A., Pătrăucean, V., Stent, S. and Cipolla, R. (2016) Scenenet: An Annotated Model Generator for Indoor Scene Understanding. 2016 IEEE International Conference on Robotics and Automation, Stockholm, 16-21 May 2016, 5737-5743. https://doi.org/10.1109/ICRA.2016.7487797
Armeni, I., Sener, O., Zamir, A.R., Jiang, H., Brilakis, I., Fischer, M. and Savarese, S. (2016) 3D Semantic Parsing of Large-Scale Indoor Spaces. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, 27-30 June 2016, 1534-1543. https://doi.org/10.1109/CVPR.2016.170
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A. and Zhang, Y. (2017) Matterport3D: Learning from RGB-D Data in Indoor Environments. 2017 International Conference on 3D Vision, Qingdao, 10-12 October 2017, 667-676. https://doi.org/10.1109/3DV.2017.00081
Eigen, D., Puhrsch, C. and Fergus, R. (2014) Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D. and Weinberger, K.Q., Eds., Proceedings of the 27th International Conference on Neural Information Processing Systems, MIT Press, Cambridge, MA, 2366-2374.
Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M. and Funkhouser, T. (2017) Semantic Scene Completion from a Single Depth Image. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 21-26 July 2017, 1746-1754. https://doi.org/10.1109/CVPR.2017.28