In this paper, a mobile assistance-system is described which supports users in performing manual working tasks in the context of assembling complex products. The assistance system contains a head-worn display for the visualization of information relevant for the workflow as well as a video camera to acquire the scene. This paper is focused on the interaction of the user with this system and describes work in progress and initial results from an industrial application scenario. We present image-based methods for robust recognition of static and dynamic hand gestures in realtime. These methods are used for an intuitive interaction with the assistance-system. The segmentation of the hand based on color information builds the basis of feature extraction for static and dynamic gestures. For the static gestures, the activation of particular sensitive regions in the camera image by the user’s hand is used for interaction. An HMM classifier is used to extract dynamic gestures depending on motion parameters determined based on the optical flow in the camera image.
KeywordsHead Mounted DisplayGesture RecognitionMobile Assistance System
Schwerdtfeger, B., Reif, R., Gunthner, W.A., Klinker, G., Hamacher, D., Schega, L., Bockelmann, I., Doil, F. and Tümler, J. (2009) Pick-by-Vision: A First Stress Test. 8th IEEE International Symposium on Mixed and Augmented Reality, Orlando, 19-22 October 2009, 115-124.
Conradi, J. and Alexander, T. (2008) Display Devices for Virtual Environments: Impact on Performance, Workload, and Simulator Sickness. EGVE 2008 Symposium Eindhoven, The Netherlands, 29-30 May 2008, 103-109.
Smith, S.P. and Hart, J. (2006) Evaluating Distributed Cognitive Resources for Way Finding in a Desktop Virtual Environment. Symposium on 3D User Interfaces (3DUI), Virginia, 25-29 March 2006, 3-10.
Gabbard, J.L., Swan, J.E., Hix, D., Kim, S.-J. and Fitch, G. (2007) Active Text Drawing Styles for Outdoor Augmented Reality: A User-Based Study and Design Implications. Virtual Reality Conference (VR), Charlotte, 10-14 March 2007, 35-42.
Tümler, J., Doil, F., Mecke, R., Paul, G., Schenk, M., Pfister, E.A., Huckauf, A., Bockelmann, I. and Roggentin, A. (2008) Mobile Augmented Reality in Industrial Applications: Approaches for Solution of User-Related Issues. 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, Cambridge, 15-18 September 2008, 87-90.
Park, M., Schmidt, L., Schlick, C. and Luczak, H. (2007) Design and Evaluation of an Augmented Reality Welding Helmet. Human Factors and Ergonomics in Manufacturing & Service Industries, 17, 317-330. http://dx.doi.org/10.1002/hfm.20077
Tümler, J., Scharf, C., Mecke, R.M., Schenk, M. and Paul, G. (2008) Incorporating User Preference to Represent Information for Manual Work Supported by Augmented Reality. SVR 2008 Symposium, João Pessoa, 13-16 May 2008, 196-203.
Fröhlich, B., Hochstrate, J., Skuk, V. and Huckauf, A. (2006) The Globefish and the Globemouse: Two New Six Degree of Freedom Input Devices for Graphics Applications. ACM CHI Conference on Human Factors in Computing Systems, Montréal, 22-27 April 2006, 191-199.
Huckauf, A., Urbina, M.H., Tümler, J., Doil, F. and Mecke, R. (2008) Visual Search in Head-Worn Displays. Perception 37 ECVP Abstract Supplement, 37, 141.
Kim, H., Albuquerque, G., Havemann, S. and Fellner, D.W. (2005) Tangible 3D: Hand Gesture Interaction for Immersive 3D Modeling. EGVE 2005 Symposium, Aalborg, 6-7 October 2005, 191-199.
Rim, B. and Schiaratura, L. (1991) Gesture and Speech. Press Syndicate of the University of Cambridge, Ch. 7, Cambridge, 239-284.
Kappas, A., Hess, U. and Scherer, K.R. (1991) Voice and Emotion. In: Feldman, R.S. and Rime, B., Eds., Fundamentals of Nonverbal Behavior, Cambridge University Press, Cambridge, 200-238.
Hasan, M.M. and Mishra, P.K. (2012) Hand Gesture Modeling and Recognition Using Geometric Features: A Review. Canadian Journal on Image Processing and Computer Vision, 3, 12-26.
Elmezain, M., Al-Hamadi, A., Rashid, O. and Michaelis, B. (2009) Posture and Gesture Recognition for Human-Computer Interaction. In-Teh, Rijeka, Ch. 23, 415-440.
Rashid, O., Al-Hamadi, A. and Michaelis, B. (2010) Utilizing Invariant Descriptors for Finger Spelling American Sign Language Using SVM. Advances in Visual Computing, 6543, 253-263.
Khan, R.Z. and Ibraheem, N.A. (2012) Hand Gesture Recognition: A Literature Review. International Journal of Artificial Intelligence & Applications (IJAIA), 3, 161-174.
Kovac, J., Peer, P. and Solina, F. (2003) Human Skin Colour Clustering for Face Detection. International Conference on Computer as a Tool EUROCON, 2, 144-148.
Cheddad, A., Condell, J., Curran, K. and Kevitt, P.M. (2009) A New Color Space for Skin Tone Detection. 16th IEEE Internation Conference on Image Processing (ICIP), Cairo, 7-10 November 2009, 497-500.
Jones, M.J. and Rehg, J.M. (1999) Statistical Color Models with Application to Skin Detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 1, 274-280.
Khan, R., Hanbury, A., Stöttinger, J. and Bais, A. (2012) Color Based Skin Classification. Pattern Recognition Letters, 33, 157-163. http://dx.doi.org/10.1016/j.patrec.2011.09.032
Ibraheem, N.A. and Khan, R. (2012) Survey on Various Gesture Recognition Technologies and Techniques. International Journal of Computer Applications, 50, 38-44.
Mittal, A., Zisserman, A. and Torr, P.H.S. (2011) Hand Detection Using Multiple Proposals. The 22nd British Machine Vision Conference, Dundee, 29 August-2 September 2011, 1-11.
Rashid, O., Al-Hamadi, A., Panning, A. and Michaelis, B. (2009) Posture Recognition Using Combined Statistical and Geometrical Feature Vectors Based on SVM. International Conference on Image, Signal and Vision Computing (ICISVC), London, 26-28 August 2009, 590-597.
Suk, H.I., Sin, B.K. and Lee, S.W. (2010) Hand Gesture Recognition Based on Dynamic Bayesian Network Framework. Pattern Recognition, 43, 3059-3072. http://dx.doi.org/10.1016/j.patcog.2010.03.016
Lange, R., Dürr, F. and Rothermel, K. (2011) Efficient Real-Time Trajectory Tracking. The VLDB Journal, 20, 671-694. http://dx.doi.org/10.1007/s00778-011-0237-7
Rekha, J., Bhattacharya, J. and Majumder, S. (2011) Hand Gesture Recognition for Sign Language: A New Hybrid Approach. The 2011 International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV), Las Vegas, 18-21 July 2011, 80-86.
Elmezain, M., Al-Hamadi, A. and Michaelis, B. (2009) A Novel System for Automatic Hand Gesture Spotting and Recognition in Stereo Color Image Sequences. Journal of WSCG, 17, 89-96.
Wagner, D., Langlotz, T. and Schmalstieg, D. (2008) Robust and Unobtrusive Marker Tracking on Mobile Phones. Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, Cambridge, 15-18 September 2008, 121-124.
Fuhrmann, A., Schmalstieg, D. and Purgathofer, W. (1999) Fast Calibration for Augmented Reality. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, London, 20-22 December 1999, 166-167.
Kato, H. and Billinghurst, M. (1999) Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. 2nd IEEE and ACM International Workshop on Augmented Reality, San Francisco, 85-94.
Tuceryan, M. and Navab, N. (2000) Single Point Active Alignment Method (SPAAM) for Optical See-Through HMD Calibration for AR. IEEE and ACM International Symposium on Augmented Reality, Munich, 5-6 October 2000, 149-158.
Lam, E.Y. (2005) Combining Gray World and Retinex Theory for Automatic White Balancein Digital Photography. 9th International Symposium on Consumer Electronics (ISCE), Macau, 14-16 June 2005, 134-139.
Phung, S.L., Bouzerdoum, A. and Chai, D. (2005) Skin Segmentation Using Color Pixel Classification: Analysis and Comparison. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 148-154. http://dx.doi.org/10.1109/TPAMI.2005.17
Farnebäck, G. (2003) Two-Frame Motion Estimation Based on Polynomial Expansion. Lecture Notes in Computer Science, 2749, 363-370.
Rabiner, L.R. (1990) A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. In: Waibel, A. and Lee, K.F., Eds., Readings in Speech Recognition, Morgan Kaufmann Publishers Inc., San Francisco, 267-296.