The present study investigated facial expressions as reactions to positive and negative feedback during a human-computer interaction. Thirty subjects interacted in natural language with a computer system while conducting a mental task, which was designed in the style of a Wizard of Oz scenario. During six different experimental sequences from a well-known memory paradigm (“Concentration”), the wizard produced different feedback as a manipulative tactic to alter the users’ reactions, e.g., praising the subject was assumed to exert a positive influence on the subjects’ moods. To analyze the impact of various feedback on the emotional reaction, facial expressions from all subjects were coded with the Facial Expression Coding System (FACES). In general, the outcomes yielded more negative facial expressions in response to positive feedback as compared with negative feedback. Furthermore, a gender difference was evident: women showed significantly more negative facial expressions in response to praise (“Your performance is improving”) than men. The study and results show that when considering future companion technologies, it is important to carefully investigate users’ reactions to feedback, since paradoxical reactions may ensue.
KeywordsAffective ComputingFacial ExpressionFACESPositive and Negative FeedbackGender Difference
Adolphs, R. (2002). Recognizing Emotion from Facial Expressions: Psychological and Neurological Mechanisms. Behavioral and Cognitive Neuroscience Reviews, 1, 21-62. http://dx.doi.org/10.1177/1534582302001001003
Bernsen, N. O., Dybkj?r, H., & Dybkj?r, L. (1994). Wizard of oz Prototyping: How and When. Paper Presented at the CCI Working Papers in Cognitive Science and HCI, Roskilde, Denmark.
Berridge, K. C. (2004). Pleasure, Unfelt Affect, and Irrational Desire. In A. S. R. Manstead, N. H. Frijda, & A. E. Fischer (Eds.), Feelings and Emotions: The Amsterdam Symposium Cambridge. Cambridge, UK: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511806582
Bosch, N., Chen, Y., & D’Mello, S. (2014). It’s Written on Your Face: Detecting Affective States from Facial Expressions While Learning Computer Programming. Paper Presented at the Intelligent Tutoring Systems. http://dx.doi.org/10.1007/978-3-319-07221-0_5
Bradley, M. M., & Lang, P. J. (1994), Measuring Emotion: the Self-Assessment Manikin and the Semantic Differential. Journal of Behavior Therapy and Experimental Psychiatry, 25, 49-59. http://dx.doi.org/10.1016/0005-7916(94)90063-9
Calvo, R., & D’Mello, S. (2010). Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Affective Computing, 1, 18-37. http://dx.doi.org/10.1109/T-AFFC.2010.1
Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S. et al. (2007). Multimodal Emotion Recognition from Expressive Faces, Body Gestures and Speech. In C. Boukis, A. Pnevmatikakis, & L. Polymenakos (Eds.), Artificial Intelligence and Innovations 2007: From Theory to Applications. Boston: Springer.
Cohen, J. (2013). Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012). Collecting Large, Richly Annotated Facial-Expression Databases from Movies. MultiMedia, IEEE, 19, 34-41. http://dx.doi.org/10.1109/MMUL.2012.26
Dickh?user, O. (2001). Computernutzung und Geschlecht. Münster und New York: Waxmann Verlag.
Ekman, P. (1992). Are There Basic Emotions? Psychological Review, 99, 550-553.
Ekman, P. (2003). Darwin, Deception, and Facial Expression. Annals of the New York Academy of Sciences, 1000, 205-221. http://dx.doi.org/10.1196/annals.1280.010
Ekman, P. (2004). Face to Face—The Science of Reading Faces. http://globetrotter.berkeley.edu/people4/Ekman/ekman-con3.html
Ekman, P., & Cordaro, D. (2011). What Is Meant by Calling Emotions Basic. Emotion Review, 3, 2364-2370. http://dx.doi.org/10.1177/1754073911410740
Ekman, P., & Friesen, W. V. (1978). Manual for the Facial Action Code. Palo Alto: Consulting Psychologist Press.
Ekman, P., & Friesen, W. V. (2003). Unmasking the Face. A Guide to Recognizing Emotions from Facial Clues. Los Altos, CA: Malor Books.
Ekman, P., Friesen, W. V., & Ellsworth, P. (2013). Can Facial Behavior Be Controlled Or Disguised? In P. Ekman, W. V. Friesen, & P. Ellsworth (Eds.), Emotion in the Human Face—Guidelines for Research and an Integration of Findings. Fairview Park: Pergamon Press Inc.
Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial Action Coding System. Manual and Investigator’s Guide. Salt Lake City, UT: Research Nexus.
Friesen, W. V., & Ekman, Pl. (1985) The Emotion Facial Action Coding System (EMFACS). Unpublished Document.
Hallgren, K. A. (2012). Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23-34.
Hoque, M. E., McDuff, D. J., & Picard, R. W. (2012). Exploring Temporal Patterns in Classifying Frustrated and Delighted Smiles. IEEE Transactions on Affective Computing, 3, 323-334. http://dx.doi.org/10.1109/T-AFFC.2012.11
Hoque, M., & Picard, R. W. (2011). Acted vs. Natural Frustration and Delight: Many People Smile in Natural Frustration. IEEE International Conference on Automatic Face & Gesture Recognition and Workshops, Santa Barbara, 21-25 Mar 2011, 354-359.
Janssen, J. H. (2012). A Three-Component Framework for Empathic Technologies to Augment Human Interaction. Journal on Multimodal User Interfaces, 6, 143-161. http://dx.doi.org/10.1007/s12193-012-0097-5
Kanade, T., Cohn, J. F., & Tian, Y. L. (2000). Comprehensive Database for Facial Expression Analysis. Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, 28-30 Mar 2000, 46-53. http://dx.doi.org/10.1109/afgr.2000.840611
Kappas, A. (2003). What Facial Activity Can and Cannot Tell Us about Emotions. In M. Katsikitis (Ed.), The Human Face (pp. 215-234). New York: Springer. http://dx.doi.org/10.1007/978-1-4615-1063-5_11
Kelley, J. F. (1984). An Iterative Design Methodology for User-Friendly Natural-Language Office Information Applications. ACM Transactions on Office Information Systems, 2, 26-41. http://dx.doi.org/10.1145/357417.357420
Keltner, D., & Anderson, C. (2000). Saving Face for Darwin the Functions and Uses of Embarrassment. Current Directions in Psychological Science, 9, 187-192. http://dx.doi.org/10.1111/1467-8721.00091
Keltner, D., & Ekman, P. (2000). Facial Expression of Emotion. In M. Lewis, & J. Haviland-Jones (Eds.), Handbook of Emotions (2nd ed., pp. 236-249). New York: Guilford Publications, Inc.
Klein, J., Moon, Y., & Picard, R. W. (2002). This Computer Responds to User Frustration: Theory, Design, and Results. Interacting with Computers, 14, 119-140. http://dx.doi.org/10.1111/1467-8721.00091
Kreisler, H. (2004). Paul Ekman Interview: Conversations with History. Berkeley, CA: Institute of International Studies, UC Berkeley. http://globetrotter.berkeley.edu/people4/Ekman/ekman-con3.html
Kring, A., & Sloan, D. (2007). The Facial Expression Coding System (FACES): Development, Validation, and Utility. Psychological Assessment, 19, 210-224. http://dx.doi.org/10.1037/1040-3590.19.2.210
Limbrecht-Ecklundt, K., Rukavina, S., Walter, S., Scheck, A., Hrabal, D., Tan, J.-W., et al. (2013). The Importance of Subtle Facial Expressions for Emotion Classification in Human-Computer Interaction. In A. Freitas-Magalh?es (Ed.), Emotional Expression: The Brain and the Face (Vol. 5). Porto: UFT Press.
Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., et al. (2011). The Computer Expression Recognition Toolbox (CERT). IEEE International Conference on Automatic Face and Gesture Recognition, Santa Barbara, 21-25 March 2011, 298-305. http://dx.doi.org/10.1109/fg.2011.5771414
Mehrabian, A., & Russell, J. A. (1974). An Approach to Environmental Psychology. Cambridge, MA: The MIT Press.
Meyer, W.-U. (1992). Paradoxical Effects of Praise and Criticism on Perceived Ability. European Review of Social Psychology, 3, 259-283. http://dx.doi.org/10.1080/14792779243000087
Partala, T., & Surakka, V. (2004). The Effects of Affective Interventions in Human-Computer Interaction. Interacting with Computers, 16, 295-309. http://dx.doi.org/10.1016/j.intcom.2003.12.001
Pentland, A. S. (2008). Honest Signals: How They Shape Our World. Cambridge, MA, London, England: MIT Press.
Pentland, A. S. (2010). To Signal Is Human. American Scientist, 98, 204-219.
Picard, R. W. (1995). Affective Computing. M.I.T Media Laboratory Perceptual Computing Section Technical Report No. 321.
Rukavina, S., Gruss, S., Walter, S., Hoffmann, H., & Traue, H. C. (2015). OPEN_EmoRec_II—A Multimodal Corpus of Human-Computer Interaction. International Journal of Computer, Electrical, Automation, Control and Information Engineering, 9, 977-983.
Russel, J. A., & Mehrabian, A. (1977). Evidence for a Three-Factor Theory of Emotions. Journal of Research in Personality, 11, 273-294. http://dx.doi.org/10.1016/0092-6566(77)90037-X
Storch, M., & Tschacher, W. (2014). Embodied Communication: Kommunikation beginnt im K?rper, nicht im Kopf. Bern: Verlag Hans Huber.
Traue, H. C., & Deighton, R. W. (2000). Emotional Inhibition. In G. Fink (Ed.), Encyclopedia of Stress. San Diego, CA: Academic Press.
Traue, H. C., & Kessler, H. (2003). Psychologische Emotionskonzepte. In A. Stephan, & H. Walter (Eds.), Natur und Theorie der Emotion (pp. 20-33). Paderborn: Mentis.
Traue, H. C., Ohl, F., Brechman, A., Schwenker, F., Kessler, H., Limbrecht, K., et al. (2013). A Framework for Emotions and Dispositions in Man-Companion Interaction. In M. Rojc, & N. Campbell (Eds.), Converbal Synchrony in Human-Machine Interaction (pp. 99-140). Boca Raton, FL: CRC Press. http://dx.doi.org/10.1201/b15477-6
Walter, S., Jonghwa, K., Hrabal, D., Crawcour, S. C., Kessler, H., & Traue, H. C. (2013). Transsituational Individual-Spe- cific Biopsychological Classification of Emotions. IEEE Transactions on Systems, Man, and Cybernetics, 43, 988-995. http://dx.doi.org/10.1109/TSMCA.2012.2216869
Walter, S., Scherer, S., Schels, M., Glodek, M., Hrabal, D., Schmidt, M., et al. (2011). Multimodal Emotion Classification in Naturalistic User Behavior. Proceedings of the 14th International Conference on Human-Computer Interaction: Towards Mobile and Intelligent Interaction Environments, Part III, Orlando, 9-14 July 2011, 603-611. http://dx.doi.org/10.1007/978-3-642-21616-9_68
Wendemuth, A., & Biundo, S. (2012). A Companion Technology for Cognitive Technical Systems. COST 2102 International Training School, Dresden, 21-26 February 2011, 89-103. http://dx.doi.org/10.1007/978-3-642-34584-5_7
Wundt, W. M. (1896). Lecture on Human and Animal Psychology. New York: Macmillan.