In this research, we study the relationship between mental workload and facial temperature of aircraft participants during a simulated takeoff flight. We conducted experiments to comprehend the correlation between work and facial temperature within the flight simulator. The experiment involved a group of 10 participants who played the role of pilots in a simulated A-320 flight. Six different flying scenarios were designed to simulate normal and emergency situations on airplane takeoff that would occur in different levels of mental workload for the participants. The measurements were workload assessment, face temperatures, and heart rate monitoring. Throughout the experiments, we collected a total of 120 instances of takeoffs, together with over 10 hours of time-series data including heart rate, workload, and face thermal images and temperatures. Comparative analysis of EEG data and thermal image types, revealed intriguing findings. The results indicate a notable inverse relationship between workload and facial muscle temperatures, as well as facial landmark points. The results of this study contribute to a deeper understanding of the physiological effects of workload, as well as practical implications for aviation safety and performance.
KeywordsMental WorkloadEEGThermal ImagesFlight SimulationAviationFace Temperature
Devos, H., Gustafson, K.M., Ahmadnezhad, P., Liao, K., Mahnken, J.D., Brooks, W.M., Burns, J.M., et al. (2020) Psychometric Properties of NASA-TLX and Index of Cognitive Activity as Measures of Cognitive Workload in Older Adults. Brain Sciences, 10, Article 994. https://doi.org/10.3390/brainsci10120994
Hananingrum, P., Athqia, A.A. and Wahyudiono, Y.D.A. (2022) Relationship between Age, Gender, Job Placement, and Social Relationships with the Mental Workload of Managers. The Indonesian Journal of Occupational Safety and Health, 11, 377-389. https://doi.org/10.20473/ijosh.v11i3.2022.377-389
Vidulich, M.A. and Tsang, P.S. (2012) Mental Workload and Situation Awareness. In: Salvendy, G., Ed., Handbook of Human Factors and Ergonomics, Wiley, New York, 243-273. https://doi.org/10.1002/9781118131350.ch8
Shah, M. and Peikari, H.R. (2016) Electronic Prescribing Usability: Reduction of Mental Workload and Prescribing Errors among Community Physicians. Telemedicine and E-Health, 22, 36-44. https://doi.org/10.1089/tmj.2014.0246
Zahednezhad, H., Shokrollahi, N., Gheshlagh, R.G. and Afshar, P.F. (2021) Does Heavy Mental Workload Affect Moral Sensitivity among Critical Care Unit Nursing Professionals? A Cross-Sectional Study. BMC Nursing, 20, Article No. 140. https://doi.org/10.1186/s12912-021-00662-8
Ganesh, K., Snekhalatha, U. and Krishnan, P.T. (2021) Deep Learning Techniques for Automated Detection of Autism Spectrum Disorder Based on Thermal Imaging. Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine, 235, 1113-1127. https://doi.org/10.1177/09544119211024778
Stemberger, J., Allison, R.S. and Schnell, T. (2010) Thermal Imaging as a Way to Classify Cognitive Workload. 2010 Canadian Conference on Computer and Robot Vision, Ontario, 31 May-2 June 2010, 231-238. https://doi.org/10.1109/CRV.2010.37
Kosonogov, V., Zorzi, L.D., Honoré, J., Martínez-Velázquez, E.S., Nandrino, J., Martínez-Selva, J.M., Sequeira, H., et al. (2017) Facial Thermal Variations: A New Marker of Emotional Arousal. PLOS ONE, 12, e0183592. https://doi.org/10.1371/journal.pone.0183592
Pollina, D.A., Dollins, A.B., Senter, S.M., Brown, T.E., Pavlidis, I., Levine, J.A., Ryan, A.H., et al. (2006) Facial Skin Surface Temperature Changes during a “Concealed Information” Test. Annals of Biomedical Engineering, 34, 1182-1189. https://doi.org/10.1007/s10439-006-9143-3
Wang, Z., Horng, G., Hsu, T., Chen, C. and Jong, G. (2020) A Novel Facial Thermal Feature Extraction Method for Non-Contact Healthcare System. IEEE Access, 8, 86545-86553. https://doi.org/10.1109/ACCESS.2020.2992908
Liu, X., Wang, Y. and Luan, J. (2021) Facial Paralysis Detection in Infrared Thermal Images Using Asymmetry Analysis of Temperature and Texture Features. Diagnostics, 11, Article 2309. https://doi.org/10.3390/diagnostics11122309
Antoine, M., Abdessalem, H.B. and Frasson, C. (2022) Cognitive Workload Assessment of Aircraft Pilots. Journal of Behavioral and Brain Science, 12, 474-484. https://doi.org/10.4236/jbbs.2022.1210027
Liu, Y., Ayaz, H. and Shewokis, P.A. (2017) Multisubject “Learning” for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures. Frontiers in Human Neuroscience, 11, Article 277607. https://doi.org/10.3389/fnhum.2017.00389
Hogervorst, M.A., Brouwer, A. and Erp, J.V. (2014) Combining and Comparing EEG, Peripheral Physiology and Eye-Related Measures for the Assessment of Mental Workload. Frontiers in Neuroscience, 8, Article 322. https://doi.org/10.3389/fnins.2014.00322
Rebsamen, B., Kwok, K. and Penney, T.B. (2011) Evaluation of Cognitive Workload from EEG during a Mental Arithmetic Task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 55, 1342-1345. https://doi.org/10.1177/1071181311551279
Rojas, R.F., Debie, E., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S.G., Abbass, H.A., et al. (2020) Electroencephalographic Workload Indicators during Teleoperation of an Unmanned Aerial Vehicle Shepherding a Swarm of Unmanned Ground Vehicles in Contested Environments. Frontiers in Neuroscience, 14, Article 40. https://doi.org/10.3389/fnins.2020.00040
Jafari, M., Zaeri, F., Jafari, A., Najafabadi, A.T.P., Al-Qaisi, S. and Hassanza-deh-Rangi, N. (2020) Assessment and Monitoring of Mental Workload in Subway Train Operations Using Physiological, Subjective, and Performance Measures. Human Factors and Ergonomics in Manufacturing & Service Industries, 30, 165-175. https://doi.org/10.1002/hfm.20831
Woods, B. and Byrne, A. (2018) The Effect of Multitasking on the Communication Skill and Clinical Skills of Medical Students. BMC Medical Education, 18, Article No. 76. https://doi.org/10.1186/s12909-018-1183-5
Lim, W.L., Sourina, O., Liu, Y. and Wang, L. (2015) EEG-Based Mental Workload Recognition Related to Multitasking. 2015 10th International Conference on Information, Communications and Signal Processing (ICICS), Singapore, 2-4 December 2015, 1-4. https://doi.org/10.1109/ICICS.2015.7459834
Han, P., Xu, L. and Lv, X. (2020) Optimised Analysis of Community Medical App User Experience under Cognitive Load Theory. E3S Web of Conferences, 179, Article ID: 02063. https://doi.org/10.1051/e3sconf/202017902063
Stone, R. and Wei, C. (2011) Exploring the Linkage between Facial Expression and Mental Workload for Arithmetic Tasks. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 55, 616-619. https://doi.org/10.1177/1071181311551126
Deng, Z., Chen, Y., Qiang, Y., Xu, Z. and Ye, X. (2023) An Experimental Study on Web Interface Design Optimization Based on User Cognitive Load. Frontiers in Artificial Intelligence and Applications, 365, 506-518. https://doi.org/10.3233/FAIA220744
Hernàndez-Sabaté, A., Vidalón, J.E.Y., Folch, P., Piera, M.À. and Gil, D. (2022) Recognition of the Mental Workloads of Pilots in the Cockpit Using EEG Signals. Applied Sciences, 12, Article 2298. https://doi.org/10.3390/app12052298
Wang, X., Li, D., Menassa, C.C. and Kamat, V.R. (2019) Can Infrared Facial Thermography Disclose Mental Workload in Indoor Thermal Environments? Proceedings of the 1st ACM International Workshop on Urban Building Energy Sensing, Controls, Big Data Analysis, and Visualization, New York, NY, 10 November 2019, 87-96. https://doi.org/10.1145/3363459.3363528
Hassoumi, A., Peysakhovich, V., Coz, A., Hurter, C. and Causse, M. (2022) Thermal Imaging of the Face: Mental Workload Detection in Flight Simulator. In: Ayaz, H. and Asgher, U. Eds., Neuroergonomics and Cognitive Engineering, Springer, Cham. https://doi.org/10.54941/ahfe1001822
Cacioppo, J.T., Tassinary, L.G. and Berntson, G. (2007) Handbook of Psychophysiology. 3rd Edition, Cambridge University Press, Cambridge.
Critchley, H. and Harrison, N.A. (2013) Visceral Influences on Brain and Behavior. Neuron, 77, 624-638. https://doi.org/10.1016/j.neuron.2013.02.008
Füstös, J., Gramann, K., Herbert, B.M. and Pollatos, O. (2012) On the Embodiment of Emotion Regulation: Interoceptive Awareness Facilitates Reappraisal. Social Cognitive and Affective Neuroscience, 8, 911-917. https://doi.org/10.1093/scan/nss089
Sokolov, A.A., Zeidman, P., Erb, M., Ryvlin, P., Friston, K.J. and Pavlova, M. (2018) Structural and Effective Brain Connectivity Underlying Biological Motion Detection. Proceedings of the National Academy of Sciences, 115, E12034-E12042. https://doi.org/10.1073/pnas.1812859115
Acharya, U.R., Joseph, K.P., Kannathal, N., Lim, C. and Suri, J.S. (2006) Heart Rate Variability: A Review. Medical and Biological Engineering and Computing, 44, 1031-1051. https://doi.org/10.1007/s11517-006-0119-0
Valenti, V.E. (2015) The Recent Use of Heart Rate Variability for Research. Journal of Human Growth and Development, 25, 137. https://doi.org/10.7322/jhgd.102991
Sztajzel, J. (2004) Heart Rate Variability: A Noninvasive Electrocardiographic Method to Measure the Autonomic Nervous System. Swiss Medical Weekly, 134, 514-522.
McCraty, R. and Shaffer, F. (2015) Heart Rate Variability: New Perspectives on Physiological Mechanisms, Assessment of Self-Regulatory Capacity, and Health Risk. Global Advances in Health and Medicine, 4, 46-61. https://doi.org/10.7453/gahmj.2014.073
Assaf, A.H., Abdessalem, H.B. and Frasson, C. (2023) Detection and Recuperation of Mental Fatigue. Journal of Behavioral and Brain Science, 13, 15-31. https://doi.org/10.4236/jbbs.2023.132002
Wang, X., Chen, J. and You, W. (2017) Feature Weight Driven Interactive Mutual Information Modeling for Heterogeneous Bio-Signal Fusion to Estimate Mental Workload. Sensors, 17, Article 2315. https://doi.org/10.3390/s17102315
Lecoutre, L., Lini, S., Bey, C., Lebour, Q. and Favier, P.A. (2015) Evaluating EEG Measures as a Workload Assessment in an Operational Video Game Setup. Proceedings of the 2nd International Conference on Physiological Computing Systems (PhyCS 2015), Angers, Loire Valley, 11-13 February 2015, 112-117.
Lowe, D.J., James, S.A., Lloyd, A. and Clegg, G. (2016) Feasibility of EEG to Monitor Cognitive Performance during Venous Cannulation: EEG Distracted Intravenous Access (e-diva). BMJ Simulation and Technology Enhanced Learning, 2, 68-72. https://doi.org/10.1136/bmjstel-2015-000082
Masaki, A., Nagumo, K., Lamsal, B., Oiwa, K. and Nozawa, A. (2020) Anomaly Detection in Facial Skin Temperature Using Variational Autoencoder. Artificial Life and Robotics, 26, 122-128. https://doi.org/10.1007/s10015-020-00634-2
Merla, A. and Romani, G.L. (2007) Thermal Signatures of Emotional Arousal: A Functional Infrared Imaging Study. 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Lyon, 22-26 August 2007, 247-249. https://doi.org/10.1109/IEMBS.2007.4352270
Gane, L., Power, S., Kushki, A. and Chau, T. (2011) Thermal Imaging of the Periorbital Regions during the Presentation of an Auditory Startle Stimulus. PLOS ONE, 6, e27268. https://doi.org/10.1371/journal.pone.0027268
Rantala, M.J., Moore, F.R., Skrinda, I., Krama, T., Kivleniece, I., Kecko, S. and Krams, I. (2012) Evidence for the Stress-Linked Immunocompetence Handicap Hypothesis in Humans. Nature Communications, 3, Article No. 694. https://doi.org/10.1038/ncomms1696
Nitschke, J.P., Sunahara, C.S., Carr, E.W., Winkielman, P., Pruessner, J.C. and Bartz, J.A. (2020) Stressed Connections: Cortisol Levels Following Acute Psychosocial Stress Disrupt Affiliative Mimicry in Humans. Proceedings of the Royal Society B: Biological Sciences, 287, Article ID: 20192941. https://doi.org/10.1098/rspb.2019.2941
Westbrook, K.E., Nessel, T.A., Hohman, M.H. and Varacallo, M. (2024) Anatomy, Head and Neck: Facial Muscles. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/29630261
Bhattacharjee, S. (2018) Procerus Sign: Mechanism, Clinical Usefulness, and Controversies. Annals of Indian Academy of Neurology, 21, 164-165. https://doi.org/10.4103/aian.AIAN_408_17
Tong, J., Lopez, M.J. and Patel, B.C. (2024) Anatomy, Head and Neck: Eye Orbicularis Oculi Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/28722936
Bloom, J., Lopez, M.J. and Rayi, A. (2024) Anatomy, Head and Neck: Eye Levator Labii Superioris Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31082075
Sobiesk, J.L. and Munakomi, S. (2024) Anatomy, Head and Neck, Nasal Cavity. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31334952
Skochdopole, A., Bay, C., Grome, L., Vorstenbosch, J., Yu, J.Z., Winocour, S., Reece, E.M., et al. (2023) Current Surgical Outcomes of Nasal Tip Grafts in Rhinoplasty: A Systematic Review. Plastic & Reconstructive Surgery, 152, 603e-616e. https://doi.org/10.1097/PRS.0000000000010257
Jain, P. and Rathee, M. (2024) Anatomy, Head and Neck, Orbicularis Oris Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31424753
Nguyen, J.D. and Duong, H. (2024) Anatomy, Head and Neck: Face. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31855374
Germann, A.M. and Al Khalili, Y. (2024) Anatomy, Head and Neck, Risorius Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31082043
Rathee, M. and Jain, P. (2024) Anatomy, Head and Neck: Buccinator Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31536284
Hoerter, J.E. and Patel, B.C. (2024) Anatomy, Head and Neck, Platysma. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/31424878
Corcoran, N.M. and Goldman, E.M. (2024) Anatomy, Head and Neck, Masseter Muscle. StatPearls, Treasure Island. https://www.ncbi.nlm.nih.gov/pubmed/30969691
Kim, J., Jeong, H., Cho, J., Pak, C.J., Oh, T.S., Hong, J.P., Yoo, J., et al. (2022) Numerical Approach to Facial Palsy Using a Novel Registration Method with 3D Facial Landmark. Sensors, 22, Article 6636. https://doi.org/10.3390/s22176636
Reineg: Muscles of the Face, Colorful Anatomy Info Poster. https://stock.adobe.com/ca/images/muscles-of-the-face-colorful-anatomy-info-poster/309366859