Traffic accidents are mainly caused by human error. In an aging society, the number of accidents attributed to elderly drivers is increasing. One noteworthy reason for this is operation misapplication. Studies have been conducted on the use of human - machine interfaces (HMIs) to inform the driver when he or she makes an error and encourage appropriate actions. However, the driver state during the erroneous action has not been investigated. The pur pose of this study is to clarify the difference in the driver’s state between normal and surprising situations in a misapplication scenario, utilizing multimodal information such as biometric information and driver operation. We found significant changes in the interaction of components between the nor mal and the surprised driving state. The results could provide basic know ledge for the future development of a driver assistance system and driver state estimation using data acquired from multiple sensors in the vehicle.
World Health Organization (2020) Road Traffic Injuries. World Health Organization, Geneva.
Klauer, S.G., Dingus, T.A., Neale, V.L., Sudweeks, J.D. and Ramsey, D.J. (2006) The Impact of Driver Inattention on Near-Crash/Crask Risk: An Analysis Using the 100-Car Naturalistic Driving Study Data. DOT HS 810594, National Highway Traffic Safety Administration, Washington DC.
Chan, M. and Singhal, A. (2013) The Emotional Side of Cognitive Distraction: Implications for Road Safety. Accident Analysis and Prevention, 50, 147-154. https://doi.org/10.1016/j.aap.2012.04.004
Gao, H., Yuce, A. and Thiran, J.-P. (2014) Detecting Emotional Stress from Facial Expressions for Driving Safety. 2014 IEEE International Conference on Image Processing, Paris, 27-30 October 2014, 5961-5965. https://doi.org/10.1109/ICIP.2014.7026203
Chan, M. and Singhal, A. (2015) Emotion Matters: Implications for Distracted Driving. Safety Science, 72, 302-309. https://doi.org/10.1016/j.ssci.2014.10.002
Scott-Parker, B. (2017) Emotions, Behaviour, and the Adolescent Driver: A Literature Review. Transportation Research Part F: Traffic Psychology and Behaviour, 50, 1-37. https://doi.org/10.1016/j.trf.2017.06.019
Rivera, J., Talone, A. B., Boesser, C. T., Jentsch, F. and Yeh, M. (2014) Startle and Surprise on the Flight Deck. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58, 1047-1051. https://doi.org/10.1177%2F1541931214581219
Landman, A., Groen, E. L., van Paassen, M. M. (René), Bronkhorst, A.W. and Mulder, M. (2017) Dealing with Unexpected Events on the Flight Deck: A Conceptual Model of Startle and Surprise. Human Factors, 59, 1161-1172. https://doi.org/10.1177%2F0018720817723428
de Boer, R.J. and Hurts, K. (2017) Automation Surprise: Results of a Field Survey of Dutch Pilots. Aviation Psychology and Applied Human Factors, 7, 28-41. https://doi.org/10.1027/2192-0923/a000113
Lococo, K.H., Staplin, L., Martell, C.A. and Sifrit, K.J. (2012) Pedal Application Errors. United States. Department of Transportation. DOT HS 811597, National Highway Traffic Safety Administration, Washington DC.
Masahide, H. (2014) Preventing Driving Operation Errors. No. 107, Institute for Traffic Accident Research and Data Analysis, Tokyo.
Akihiro, H. (2018) Accidents Due to the Misapplication of Accelerator and Brake Pedals. No. 124, Institute for Traffic Accident Research and Data Analysis, Tokyo.
D’Orazio, T., Leo, M., Guaragnella, C. and Distante, A. (2007) A Visual Approach for Driver Inattention Detection. Pattern Recognition, 40, 2341-2355. https://doi.org/10.1016/j.patcog.2007.01.018
Fletcher, L. and Zelinsky, A. (2009) Driver Inattention Detection Based on Eye Gaze—Road Event Correlation. The International Journal of Robotics Research, 28, 774-801. https://doi.org/10.1177%2F0278364908099459
Bergasa, L.M., Almeria, D., Almazan, J., Yebes, J.J. and Arroyo, R. (2014) DriveSafe: An App for Alerting Inattentive Drivers and Scoring Driving Behaviors. 2014 IEEE Intelligent Vehicles Symposium Proceedings, Dearborn, 8-11 June 2014, 240-245. https://doi.org/10.1109/IVS.2014.6856461
Liao, Y., Li, S.E., Wang, W., Wang, Y., Li, G. and Cheng, B. (2016) Detection of Driver Cognitive Distraction: A Comparison Study of Stop-Controlled Intersection and Speed-Limited Highway. IEEE Transactions on Intelligent Transportation Systems, 17, 1628-1637. https://doi.org/10.1109/TITS.2015.2506602
Gershon, P., Sita, K.R., Zhu, C., Ehsani, J.P., Klauer, S.G., Dingus, T.A. and Simons-Morton, B.G. (2019) Distracted Driving, Visual Inattention, and Crash Risk Among Teenage Drivers. American Journal of Preventive Medicine, 56, 494-500. https://doi.org/10.1016/j.amepre.2018.11.024
Lee, K.W., Yoon, H.S., Song, J.M. and Park, K.R. (2018) Convolutional Neural Network-Based Classification of Driver’s Emotion during Aggressive and Smooth Driving Using Multi-Modal Camera Sensors. Sensors, 18, 957. https://doi.org/10.3390/s18040957
Grimm, M., Kroschel, K., Harris, H., Nass, C., Schuller, B., Rigoll, G. and Moosmayr, T. (2007) On the Necessity and Feasibility of Detecting a Driver’s Emotional State While Driving. International Conference on Affective Computing and Intelligent Interaction, Lisbon, 12-14 September 2007, 126-138. https://doi.org/10.1007/978-3-540-74889-2_12
Rebolledo-Mendez, G., Reyes, A., Paszkowicz, S., Land, J., Limited, R., Reyes, A., Domingo, M.C. and Skrypchuk, L. (2014) Developing a Body Sensor Network to Detect Emotions During Driving. IEEE Transactions on Intelligent Transportation Systems, 15, 1850-1854. https://doi.org/10.1109/TITS.2014.2335151
Verma, B. and Choudhary, A. (2018) A Framework for Driver Emotion Recognition Using Deep Learning and Grassmann Manifolds. 2018 21st International Conference on Intelligent Transportation Systems, Maui, 4-7 November 2018, 1421-1426. https://doi.org/10.1109/ITSC.2018.8569461
Wu, Y., Boyle, L.N. and McGehee, D. (2018) Evaluating Variability in Foot to Pedal Movements Using Functional Principal Components Analysis. Accident Analysis and Prevention, 118, 146-153. https://doi.org/10.1016/j.aap.2018.02.011
Wu, Y., Boyle, L.N., McGehee, D., Roe, C.A., Ebe, K. and Foley, J. (2017) Foot Placement during Error and Pedal Applications in Naturalistic Driving. Accident Analysis and Prevention, 99, 102-109. https://doi.org/10.1016/j.aap.2016.10.019
Hasegawa, K., Kimura, M. and Takeda, Y. (2020) Pedal Misapplication: Interruption Effects and Age-Related Differences. Human Factors: The Journal of the Human Factors and Ergonomics Society, 001872082093612. https://doi.org/10.1177%2F0018720820936122
Schmidt, R.A., Young, D.E., Ayres, T.J. and Wong, J.R. (1997) Pedal Misapplications: Their Frequency and Variety Revealed through Police Accident Reports. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 41, 1023-1027. https://doi.org/10.1177%2F107118139704100266
Green, M. (2000) “How Long Does It Take to Stop?” Methodological Analysis of Driver Perception-Brake Times. Transportation Human Factors, 2, 195-216. https://doi.org/10.1207/STHF0203_1
Fitch, G.M., Blanco, M., Morgan, J.F. and Wharton, A.E. (2012) Driver Braking Performance to Surprise and Expected Events. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54, 2075-2080. https://doi.org/10.1177%2F154193121005402412
Freund, B., Colgrove, L.A., Petrakos, D. and McLeod, R. (2008) In My Car the Brake Is on the Right: Pedal Errors among Older Drivers. Accident Analysis & Prevention, 40, 403-409. https://doi.org/10.1016/j.aap.2007.07.012
Boer, E.R. (2001) Behavioral Entropy as a Measure of Driving Performance. Proceedings of the 1st International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Colorado, 14-17 August 2001, 226-229. https://doi.org/10.17077/drivingassessment.1043
Choi, J. and Gutierrez-Osuna, R. (2009) Using Heart Rate Monitors to Detect Mental Stress. 2009 6th International Workshop on Wearable and Implantable Body Sensor Networks, Berkeley, 3-5 June 2009, 219-223. https://doi.org/10.1109/BSN.2009.13
Solovey, E.T., Zec, M., Garcia Perez, E.A., Reimer, B. and Mehler, B. (2014) Classifying Driver Workload Using Physiological and Driving Performance Data. Proceedings of the 32nd annual ACM conference on Human Factors in computing systems, Toronto, April 2014, 4057-4066. https://doi.org/10.1145/2556288.2557068
Hart, S.G. (2006) NASA-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society, 50, 904-908. https://doi.org/10.1177%2F154193120605000909
Pickup, L., Wilson, J.R., Norris, B.J., Mitchell, L. and Morrisroe, G. (2005) The Integrated Workload Scale (IWS): A New Self-Report Tool to Assess Railway Signaller Workload. Applied Ergonomics, 36, 681-693. https://doi.org/10.1016/j.apergo.2005.05.004
Dahlhaus, R. and Eichler, M. (2003) Causality and Graphical Models in Time Series Analysis. In: Green, P., Hjort, N. and Richardson, S., Eds., Highly Structured Stochastic Systems, Oxford University Press, Oxford, 115-137.
Epskamp, S., Waldorp, L.J., Mõttus, R. and Borsboom, D. (2018) The Gaussian Graphical Model in Cross-Sectional and Time-Series Data. Multivariate Behavioral Research, 53, 453-480. https://doi.org/10.1080/00273171.2018.1454823
Tsujita, S., Azuma, A., Ikoma, T., Aoki, K., Hirano, A., Son, L.A., Nguyen, T.T., Shimazaki, K. and Aoki, H. (2019) Effects of the HMI on Elderly Drivers’ Behavior under a Simulated Unexpected Situation. 2019 JSAE Annual Congress, Yokohama, 22-24 May 2019.
Vicente, J., Laguna, P., Bartra, A. and Bailón, R. (2016) Drowsiness Detection Using Heart Rate Variability. Medical & Biological Engineering & Computing, 54, 927-937. https://doi.org/10.1007/s11517-015-1448-7
Castaldo, R., Montesinos, L., Melillo, P., James, C. and Pecchia, L. (2019) Ultra-Short Term HRV Features as Surrogates of Short Term HRV: A Case Study on Mental Stress Detection in Real Life. BMC Medical Informatics and Decision Making, 19, Article No. 12. https://doi.org/10.1186/s12911-019-0742-y
Pecchia, L., Castaldo, R., Montesinos, L. and Melillo, P. (2018) Are Ultra-Short Heart Rate Variability Features Good Surrogates of Short-Term Ones? State-of-the-Art Review and Recommendations. Healthcare Technology Letters, 5, 94-100. https://doi.org/10.1049/htl.2017.0090
Engström, J., Markkula, G., Keinath, A., Horst, D., Juch, S., Mattes, S., Foehl, U., Östlund, J., Peters, B. and Thorslund, B. (2005) Driving Performance Assessment— Methods and Metrics.
Laska, J. and Narayan, M. (2017) Skggm 0.2.7: A Scikit-Learn Compatible Package for Gaussian and Related Graphical Models. https://doi.org/10.5281/zenodo.830033
Pedregosa, F., Weiss, R., Brucher, M., Varoquaux, G., Gramfort, A., Michel, V., et al. (2011) Scikit-Learn: Machine Learning in Python. Journal of Machine Learning Research, 12, 2825-2830.
Nomikos, M.S., Opton, E. and Averill, J.R. (1968) Surprise versus Suspense in the Production of Stress Reaction. Journal of Personality and Social Psychology, 8, 204-208. https://doi.apa.org/doi/10.1037/h0025274
Gantiva, C., Araujo, A., Calderón, L., Gómez, P. and Reina, F. (2019) Psychophysiological Responses to Facial Expression of Surprise, Sadness, and Disgust. Australian Journal of Psychology, 71, 100-107. https://doi.org/10.1111/ajpy.12222
McDonald, A.D., Ferris, T.K. and Wiener, T.A. (2020) Classification of Driver Distraction: A Comprehensive Analysis of Feature Generation, Machine Learning, and Input Measures. Human Factors, 62, 1019-1035. https://doi.org/10.1177%2F0018720819856454
Forster, Y., Naujoks, F., Neukum, A. and Huestegge, L. (2017) Driver Compliance to Take-over Requests with Different Auditory Outputs in Conditional Automation. Accident Analysis and Prevention, 109, 18-28. https://doi.org/10.1016/j.aap.2017.09.019
Kozak, K., Pohl, J., Birk, W., Greenberg, J., Artz, B., Blommer, M., Cathey, L. and Curry, R. (2006) Evaluation of Lane Departure Warnings for Drowsy Drivers. In Proceedings of the Human Factors and Ergonomics Society, 20, 2400-2404. https://doi.org/10.1177%2F154193120605002211
Campbell, J.L., Richard, C.M., Brown, J.L. and McCallum, M. (2007) Crash Warning System Interfaces: Human Factors Insight and Lessons Learned. DOT HS-810, National Highway Traffic Safety Administration, Washington DC.