Background : Children with autism spectrum disorder (ASD) show impairment in producing facial expressions adapted to social contexts. Several serious games have been computed to help them dealing with facial expression recognition but very few focused on facial expression production adapted to a given social context. Method: JEMImE is a new serious game which aims to help the player to learn how to produce happiness, anger and sadness in a 3D virtual environment with social situations that should be resolved by producing the correct facial expression. He is guided in the game thanks to facial expression feedback and gauges that help him evaluating the quality of his/her production in real time. The feedbacks on the children productions are timely given by a facial expression recognition algorithm integrated in JEMImE architecture. Specific attention was paid to the visual and motivational aspects of the game. Using a brief feasibility study with children with ASD (N = 23), we evaluated the impression of the players on the game aspect and the possibility to insert algorithmic feedbacks in real time inside JEMImE. Results: During the training phase, children with ASD showed a significant progression during training for facial expression production after algorithmic autonomous feedbacks. This means children understand the challenge and that the algorithmic feedbacks are transparent enough to allow gaming. They expressed an overall good subjective experience with JEMImE in terms of ergonomics, playability, visual aspect and motivation. Conclusion: We conclude that the beta-version of JEMImE shows promising potential and that research should proceed on computing more games and scenarios to offer a longer game exposure to children to allow adequate clinical validation.
KeywordsAutism Spectrum DisorderInformation Communication TechnologiesFacial Expressions ProductionSerious GamesFacial Recognition Algorithm
Bennett, D. S., Bendersky, M., & Lewis, M. (2005). Does the Organization of Emotional Expression Change over Time? Facial Expressivity from 4 to 12 Months. Infancy, 8, 167-187. https://doi.org/10.1207/s15327078in0802_4
Bono, V., Narzisi, A., Jouen, A. L., Tilmont, E., Hommel, S., Jamal, W., Chetouani, M. et al. (2016). GOLIAH: A Gaming Platform for Home-Based Intervention in Autism-Principles and Design. Frontiers in Psychiatry, 7, 70. https://doi.org/10.3389/fpsyt.2016.00070
Boucenna, S., Narzisi, A., Tilmont, E., Muratori, F., Pioggia, G., Cohen, D., & Chetouani, M. (2014). Interactive Technologies for Autistic Children: A Review. Cognitive Computation, 6, 722-740. https://doi.org/10.1007/s12559-014-9276-x
Brun, P. (2001). Psychopathologie de l’émotion chez l’enfant: L’importance des données développementales typiques. Enfance, 53, 281-291. https://doi.org/10.3917/enf.533.0281
Chaplin, T. M., & Aldao, A. (2013). Gender Differences in Emotion Expression in Children: A Meta-Analytic Review. Psychological Bulletin, 139, 735. https://doi.org/10.1037/a0030737
Cockburn, J., Bartlett, M., Tanaka, J., Movellan, J., Pierce, M., & Schultz, R. (2008). Smilemaze: A Tutoring System in Real-Time Facial Expression Perception and Production in Children with Autism Spectrum Disorder. In ECAG 2008 Workshop Facial and Bodily Expressions for Control and Adaptation of Games (Vol. 3). Amsterdam.
Dapogny, A., Bailly, K., & Dubuisson, S. (2015). Pairwise Conditional Random Forests for Facial Expression Recognition. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3783-3791). https://doi.org/10.1109/ICCV.2015.431
Ekman, P., Roper, G., & Hager, J. C. (1980). Deliberate Facial Movement. Child Development, 886-891. https://doi.org/10.2307/1129478
Faso, D. J., Sasson, N. J., & Pinkham, A. E. (2015). Evaluating Posed and Evoked Facial Expressions of Emotion from Adults with Autism Spectrum Disorder. Journal of Autism and Developmental Disorders, 45, 75-89. https://doi.org/10.1007/s10803-014-2194-7
Fernandes, T., Alves, S., Miranda, J., Queirós, C., & Orvalho, V. (2011). LIFEisGAME: A Facial Character Animation System to Help Recognize Facial Expressions. In International Conference on Enterprise Information Systems (pp. 423-432). Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-642-24352-3_44
Grossard, C., Chaby, L., Hun, S., Pellerin, H., Bourgeois, J., Dapogny, A., Chen, L. et al. (2018). Children Facial Expression Production: Influence of Age, Gender, Emotion Subtype, Elicitation Condition and Culture. Frontiers in Psychology, 9, 446. https://doi.org/10.3389/fpsyg.2018.00446
Grossard, C., Grynspan, O., Serret, S., Jouen, A. L., Bailly, K., & Cohen, D. (2017). Serious Games to Teach Social Interactions and Emotions to Individuals with Autism Spectrum Disorders (ASD). Computers & Education, 113, 195-211. https://doi.org/10.1016/j.compedu.2017.05.002
Grossman, R. B., Edelson, L. R., & Tager-Flusberg, H. (2013). Emotional Facial and Vocal Expressions during Story Retelling by Children and Adolescents with High-Functioning Autism. Journal of Speech, Language, and Hearing Research, 56, 1035-1044. https://doi.org/10.1044/1092-4388(2012/12-0067)
Grynszpan, O., Weiss, P. L., Perez-Diaz, F., & Gal, E. (2014). Innovative Technology-Based Interventions for Autism Spectrum Disorders: A Meta-Analysis. Autism, 18, 346-361. https://doi.org/10.1177/1362361313476767
Jones, C. R., Pickles, A., Falcaro, M., Marsden, A. J., Happé, F., Scott, S. K., Simonoff, E. et al. (2011). A Multimodal Approach to Emotion Recognition Ability in Autism Spectrum Disorders. Journal of Child Psychology and Psychiatry, 52, 275-285. https://doi.org/10.1111/j.1469-7610.2010.02328.x
Josman, N., Ben-Chaim, H. M., Friedrich, S., & Weiss, P. L. (2008). Effectiveness of Virtual Reality for Teaching Street-Crossing Skills to Children and Adolescents with Autism. International Journal on Disability and Human Development, 7, 49-56. https://doi.org/10.1515/IJDHD.2008.7.1.49
Langdell, T. (1981). Face Perception: An Approach to the Study of Autism. Doctoral Dissertation, London: University of London. http://discovery.ucl.ac.uk/1317720/
Loveland, K. A., Tunali-Kotoski, B., Pearson, D. A., Brelsford, K. A., Ortegon, J., & Chen, R. (1994). Imitation and Expression of Facial Affect in Autism. Development and Psychopathology, 6, 433-444. https://doi.org/10.1017/S0954579400006039
Mitchell, P., Parsons, S., & Leonard, A. (2007). Using Virtual Environments for Teaching Social Understanding to 6 Adolescents with Autistic Spectrum Disorders. Journal of Autism and Developmental Disorders, 37, 589-600. https://doi.org/10.1007/s10803-006-0189-8
Oster, H. (2005). The Repertoire of Infant Facial Expressions: An Ontogenetic Perspective. Emotional Development, 261-292.
Park, J. H., Abirached, B., & Zhang, Y. (2012). A Framework for Designing Assistive Technologies for Teaching Children with ASDs Emotions. In CHI’12 Extended Abstracts on Human Factors in Computing Systems (pp. 2423-2428). New York: ACM. https://doi.org/10.1145/2212776.2223813
Senju, A. (2013). Atypical Development of Spontaneous Social Cognition in Autism Spectrum Disorders. Brain and Development, 35, 96-101. https://doi.org/10.1016/j.braindev.2012.08.002
Serret, S., Hun, S., Iakimova, G., Lozada, J., Anastassova, M., Santos, A., Askenazy, F. et al. (2014). Facing the Challenge of Teaching Emotions to Individuals with Low- and High-Functioning Autism Using a New Serious Game: A Pilot Study. Molecular Autism, 5, 37. https://doi.org/10.1186/2040-2392-5-37
Sullivan, M. W., & Lewis, M. (2003). Emotional Expressions of Young Infants and Children: A Practitioner’s Primer. Infants & Young Children, 16, 120-142. https://doi.org/10.1097/00001163-200304000-00005
Tan, C. T., Harrold, N., & Rosser, D. (2013). Can you CopyMe?: An Expression Mimicking Serious Game. In SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications (p. 73). ACM. https://doi.org/10.1145/2543651.2543687
Trevisan, D. A., Hoskyn, M., & Birmingham, E. (2018). Facial Expression Production in Autism: A Meta-Analysis. Autism Research, 11, 1586-1601. https://doi.org/10.1002/aur.2037
Vannetzel, L., Chaby, L., Cautru, F., Cohen, D., & Plaza, M. (2011). Neutral versus Emotional Human Stimuli Processing in Children with Pervasive Developmental Disorders Not Otherwise Specified. Research in Autism Spectrum Disorders, 5, 775-783. https://doi.org/10.1016/j.rasd.2010.09.005
Volker, M. A., Lopata, C., Smith, D. A., & Thomeer, M. L. (2009). Facial Encoding of Children with High-Functioning Autism Spectrum Disorders. Focus on Autism and Other Developmental Disabilities, 24, 195-204. https://doi.org/10.1177/1088357609347325
Yin, L., Sun, X. C. Y., Worm, T., & Reale, M. (2008). A High-Resolution 3D Dynamic Facial Expression Database. In IEEE International Conference on Automatic Face and Gesture Recognition (Vol. 126). Amsterdam, The Netherlands. https://doi.org/10.1109/AFGR.2008.4813324
Yirmiya, N., Kasari, C., Sigman, M., & Mundy, P. (1989). Facial Expressions of Affect in Autistic, Mentally Retarded and Normal Children. Journal of Child Psychology and Psychiatry, 30, 725-735. https://doi.org/10.1111/j.1469-7610.1989.tb00785.x
Yusoff, A. (2010). A Conceptual Framework for Serious Games and Its Validation. Doctoral Dissertation, Southampton: University of Southampton. https://eprints.soton.ac.uk/267690/1/3711a021.pdf https://doi.org/10.1109/ICALT.2009.19