Usability Experiments to Evaluate UML/SysML-Based Model Driven Software Engineering Notations for Logic Control in Manufacturing Automation — Oak Academic Publishing
Research ArticleOpen AccessGoogle Scholar indexed
Usability Experiments to Evaluate UML/SysML-Based Model Driven Software Engineering Notations for Logic Control in Manufacturing Automation
Institute of Automation and Information Systems, Technische Universitat München, Munich, Germany
1 Institute of Automation and Information Systems, Technische Universitat München, Munich, Germany
Many industrial companies and researchers are looking for more efficient model driven engineering approaches (MDE) in software engineering of manufacturing automation systems (MS) especially for logic control programming, but are uncertain about the applicability and effort needed to implement those approaches in comparison to classical Programmable Logic Controller (PLC) programming with IEC 61131-3. The paper summarizes results of usability experiments evaluating UML and SysML as software engineering notations for a MDE applied in the domain of manufacturing systems. Modeling MS needs to cover the domain specific characteristics, i.e. hybrid process, real time requirements and communication requirements. In addition the paper presents factors, constraint and practical experience for the development of further usability experiments. The paper gives examples of notational expressiveness and weaknesses of UML and SysML. The appendix delivers detailed master models, representing the correct best suited model, and evaluation schemes of the experiment, which is helpful if setting up own empirical experiments.
KeywordsManufacturing Automation SoftwareProgramming EnvironmentSoftware EngineeringUser Interface Human Factors
Basile, F., Chiaccio, P. and Gerbasio, D. (2012) On the Implementation of Industrial Automation Systems Based on PLC. IEEE Transactions on Automation Science and Engineering, 10, 990-1003. http://dx.doi.org/10.1109/TASE.2012.2226578
International Electrotechnical Commission (2013) IEC International Standard IEC 61131-3: Programmable Logic Controllers, Part 3: Programming Languages. IEC, Geneva.
Thramboulidis, K. (2010) The 3+1 SysML View-Model in Model Integrated Mechatronics. Journal of Software Engineering & Applications, 3, 109-118. http://dx.doi.org/10.4236/jsea.2010.32014
Rzevski, G. (2003) On Conceptual Design of Intelligent Mechatronic Systems. Mechatronics, 13, 1029-1044. http://dx.doi.org/10.1016/S0957-4158(03)00041-2
Zhabelova, G. and Vyatkin, V. (2012) Multiagent Smart Grid Automation Architecture Based on IEC 61850/61499 Intelligent Logical Nodes. IEEE Transactions on Industrial Electronics, 59, 2351-2362. http://dx.doi.org/10.1109/TIE.2011.2167891
Sauter, T. and Lobashov, M. (2011) End-to-End Communication Architecture for Smart Grids. IEEE Transactions on Industrial Electronics, 58, 1218-1228. http://dx.doi.org/10.1109/TIE.2010.2070771
Vyatkin, V. (2013) Software Engineering in Industrial Automation: State-of-the-Art Review. IEEE Transactions on Industrial Informatics, 9, 1234-1249. http://dx.doi.org/10.1109/TII.2013.2258165
Yang, C. and Vyatkin, V. (2012) Transformation of Simulink Models to IEC 61499 Function Blocks for Verification of Distributed Control Systems. Control Engineering Practice, 20, 1259-1269. http://dx.doi.org/10.1016/j.conengprac.2012.06.008
Dubinin, V., Vyatkin, V. and Pfeiffer, T. (2005) Engineering of Validatable Automation Systems Based on an Extension of UML Combined with Function Blocks of IEC 61499. IEEE International Conference on Robotics and Automation (ICRA), Barcelona, 18-22 April 2005, 3996-4001.
Secchi, C., Bonfé, M. and Fantuzzi, C. (2007) On the Use of UML for Modeling Mechatronic Systems. IEEE Transactions on Automation Science and Engineering, 4, 105-113. http://dx.doi.org/10.1109/TASE.2006.879686
Bassi, L., Secchi, C., Bonfé, M. and Fantuzzi, C. (2011) A SysML-Based Methodology for Manufacturing Machinery Modeling and Design. IEEE/ASME Transactions on Mechatronics, 16, 1049-1062. http://dx.doi.org/10.1109/TMECH.2010.2073480
Bonfé, M., Fantuzzi, C. and Secchi, C. (2013) Design Patterns for Model-Based Automation Software Design and Implementation. Control Engineering Practice, 21, 1608-1619. http://dx.doi.org/10.1016/j.conengprac.2012.03.017
Thramboulidis, K. and Frey, G. (2011) Towards a Model-Driven IEC 61131-Based Development Process in Industrial Automation. Journal of Software Engineering and Applications, 4, 217-226. http://dx.doi.org/10.4236/jsea.2011.44024
Thramboulidis, K. (2012) IEC 61131 as Enabler of OO and MDD in Industrial Automation. IEEE International Conference on Industrial Informatics (INDIN), Beijing, 25-27 July 2012, 425-430.
Obermeier, M., Braun, S. and Vogel-Heuser, B. (2014) A Model Driven Approach on Object Oriented PLC Programming for Manufacturing Systems with Regard to Usability. IEEE Transactions on Industrial Informatics, 1. http://dx.doi.org/10.1109/TII.2014.2346133
Vogel-Heuser, B., Schütz, D., Timo, F. and Legat, C. (2014) Model-Driven Engineering of Manufacturing Automation Software Projects—A SysML-Based Approach. Mechatronics, 24, 883-897. http://dx.doi.org/10.1016/j.mechatronics.2014.05.003
Estévez, E. and Marcos, M. (2012) Model-Based Validation of Industrial Control Systems. IEEE Transactions on Industrial Informatics, 8, 302-310. http://dx.doi.org/10.1109/TII.2011.2174248
Estévez, E., Marcos, M., Iriondo, N. and Orive, D. (2007) Graphical Modeling of PLC-Based Industrial Control Applications. Proceedings of the 2007 American Control Conference, New York, 11-13 July 2007, 220-225.
Bartels, J. and Vogel, B. (2001) System Engineering Approach for Plant Automation (Systementwicklung für die Automatisierung im Anlagenbau). At-Automatisierungstechnik, 49, 214-224. http://dx.doi.org/10.1524/auto.2001.49.5.214
Land, M. and Horwood, J. (1995) Which Parts of the Road Guide Steering? Nature, 377, 339-340. http://dx.doi.org/10.1038/377339a0
Savioja, P. and Norros, L. (2013) Systems Usability Framework for Evaluating Tools in Safety-Critical Work. Cognition, Technology & Work, 15, 255-275. http://dx.doi.org/10.1007/s10111-012-0224-9
Siau, K. and Rossi, M. (2011) Evaluation Techniques for Systems Analysis and Design Modeling Methods—A Review and Comparative Analysis. Information Systems Journal, 21, 249-268. http://dx.doi.org/10.1111/j.1365-2575.2007.00255.x
Katzke, U., Vogel-Heuser, B. and Fischer, K. (2004) Analysis and State of the Art of Modules in Industrial Automation. ATP International-Automation Technology in Practice International, 46, 23-31.
Nielsen, J. (1993) Usability Engineering. Academic Press, Boston.
Obermeier, M., Schütz, D. and Vogel-Heuser, B. (2012) Evaluation of a Newly Developed Model-Driven PLC Programming Approach for Machine and Plant Automation. 8th IEEE International Conference on Systems, Man and Cybernetics (SMC), Seoul, 14-17 October 2012, 1552-1557.
Frank, U., Papenfort, J. and Schütz, D. (2011) Real-Time Capable Software Agents on IEC 61131 Systems-Developing a Tool Supported Method. Proceedings of 18th IFAC World Congress, Milan, 28 August-2 September 2011, 9164-9169.
Str?mman, M., Sierla, S. and Koskinen, K. (2005) Control Software Reuse Strategies with IEC 61499. 10th IEEE International Conference on Emerging Technologies & Factory Automation (ETFA), Catania, 19-22 September 2005, 749-756.
Sierla, S., Christensen, J., Koskinen, K. and Peltola, J. (2007) Educational Approaches for the Industrial Acceptance of IEC 61499. IEEE International Conference on Emerging Technologies & Factory Automation (ETFA), Patras, 25-28 September 2007, 482-489.
Patig, S. (2008) Preparing Meta-Analysis of Metamodel Understandability. Workshop on Empirical Studies of Model-Driven Engineering (ESMDE 2008), Toulouse, 29 September 2008, 11-20.
Patig, S. (2008) A Practical Guide to Testing the Understandability of Notations. Proceedings of the Fifth Asia-Pacific Conference on Conceptual Modelling, Wollongong, 79, 49-58.
Gemino, A. and Wand, Y. (2004) A Framework for Empirical Evaluation of Conceptual Modeling Techniques. Requirements Engineering, 9, 248-260. http://dx.doi.org/10.1007/s00766-004-0204-6
Vogel-Heuser, B. and Sommer, K. (2011) A Methodological Approach to Evaluate the Benefit and Usability of Different Modeling Notations for Automation Systems. Proceedings of the 7th IEEE International Conference on Automation Science and Engineering (CASE), Trieste, 24-27 August 2011, 474-481.
Lucas, M.R. and Tilbury, D.M. (2005) Methods of Measuring the Size and Complexity of PLC Programs in Different Logic Control Design Methodologies. The International Journal of Advanced Manufacturing Technology, 26, 436-447. http://dx.doi.org/10.1007/s00170-003-1996-0
Frey, G., Litz, L. and Kl?ckner, F. (2000) Complexity Metrics for Petri Net Based Logic Control Algorithms. IEEE International Conference on Systems, Man, and Cybernetics, Nashville, 2, 1204-1209.
Venkatesh, K., Zhou, M. and Caudill, R.J. (1994) Comparing Ladder Logic Diagrams and Petri Nets for Sequence Controller Design through a Discrete Manufacturing System. IEEE Transactions on Industrial Electronics, 41, 611-619. http://dx.doi.org/10.1109/41.334578
Lee, J.S. and Hsu, P.L. (2001) A New Approach to Evaluate Ladder Diagrams and Petri Nets via the IF-THEN Transformation. IEEE International Conference on Systems, Man and Cybernetics, Tucson, 7-10 October 2001, 2711-2716.
Chidamber, S.R. and Kemerer, C.F. (1994) A Metrics Suite for Object Oriented Design. IEEE Transactions on Software Engineering, 20, 476-493. http://dx.doi.org/10.1109/32.295895
Michura, J. and Capretz, M.A.M. (2005) Metrics Suite for Class Complexity. IEEE International Conference on Information Technology: Coding and Computing (ITCC), Las Vegas, 4-6 April 2005, 404-409.
Fuchs, J., Feldmann, S., Legat, C. and Vogel-Heuser, B. (2014) Identification of Design Patterns for IEC 61131-3 in Machine and Plant Manufacturing. 19th IFAC World Congress, Cape Town, 24-29 August 2014, 6092-6097.
Park, E., Tilbury, D.M. and Khargonekar, P.P. (2001) A Modeling and Analysis Methodology for Modular Logic Controllers of Machining Systems Using Petri Net Formalism. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 31, 168-188. http://dx.doi.org/10.1109/5326.941841
Fuchs, J. (2011) Analyse und Neukonzeption der Softwaremodularit?t und deren Abbildung auf die maschinenbauliche Modularit?t am Beispiel eines Neuglasabschiebers der Lebensmittelindustrie. B.S. Thesis, Faculty of Mechanical engineering, TUM, Munich.
Recker, J.C., zur Muehlen, M., Keng, S., Erickson, J. and Indulska, M. (2009) Measuring Method Complexity: UML versus BPMN. Proceedings of the 15th Americas Conference on Information Systems, San Francisco, 6-9 August 2009, 1-10.
Rossi, M. and Brinkkemper, S. (1996) Complexity Metrics for Systems Development Methods and Techniques. Information Systems, 21, 209-227. http://dx.doi.org/10.1016/0306-4379(96)00012-9
Schalles, C. (2012) Usability Evaluation of Modeling Languages. Ph.D. Dissertation, Department of Computing, CIT, Cork.
Frank, T. (2014) Entwicklung und Evaluation einer Modellierungssprache für den Architekturentwurf von verteilten Automatisierungsanlagen auf Basis der Systems Modeling Language (SysML). Ph.D. Dissertation, Institute of Automation and Information Systems, Technical University Munich, Munich.
Hajarnavis, V. and Young, K. (2008) An Assessment of PLC Software Structure Suitability for the Support of Flexible Manufacturing Processes. IEEE Transactions on Automation Science and Engineering, 5, 641-650. http://dx.doi.org/10.1109/TASE.2007.917135
Cross, J. and Denning, P. (2001) Computing Curriculum 2001. The Joint Curriculum Task Force IEEE-CS/ACM Report. http://www.acm.org/education/curric_vols/cc2001.pdf
Tucker, A., Deek, F., Jones, J., McCowan, D., Stephenson, C. and Verno, A. (2003) A Model Curriculum for K-12 Computer Science: Final Report of the ACM K-12 Task Force Curriculum Committee. The Association for Computing Machinery, New York.
Fitzgerald, S., McCauley, R., Hanks, B., Murphy, L., Simon, B. and Zander, C. (2010) Debugging from the Student Perspective. IEEE Transactions on Education, 53, 390-396. http://dx.doi.org/10.1109/TE.2009.2025266
Curtis, B. (1988) Five Paradigms in the Psychology of Programming. In: Helander, M., Ed., Handbook of Human-Computer Interaction, Elsevier North Holland, Amsterdam, 87-105. http://dx.doi.org/10.1016/B978-0-444-70536-5.50009-9
Magenheim, J., Nelles, W., Rhode, T., Schaper, N., Schubert, S. and Stechert, P. (2010) Competencies for Informatics Systems and Modeling: Results of Qualitative Content Analysis of Expert Interviews. IEEE Education Engineering, Madrid, 14-16 April 2010, 513-521.
Ruocco, A.S. (2003) Experiences in Threading UML throughout a Computer Science Program. IEEE Transactions on Education, 46, 226-228. http://dx.doi.org/10.1109/TE.2002.808263
Wickens, C.D. and Hollands, J.G. (2000) Engineering Psychology and Human Performance. 3rd Edition, Prentice Hall, Upper Saddle River.
Schweizer, K., Gram?, D., Mühlhausen, S. and Vogel-Heuser, B. (2009) Mental Models in Process Visualization— Could They Indicate the Effectiveness of an Operator’s Training? Engineering Psychology and Cognitive Ergonomics, Springer, Berlin, Heidelberg, 297-306.
Gravetter, F.J. and Wallnau, L.B. (2006) Statistics for the Behavioral Sciences. Thomson/Wadsworth, Belmont.
Kim, J. and Lerch, F.J. (1992) Towards a Model of Cognitive Process in Logical Design: Comparing Object-Oriented and Traditional Functional Decomposition Software Methodologies. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Monterey, 3-7 July 1992, 489-498.
Kim, S.H. and Jeon, J.W. (2009) Introduction for Freshmen to Embedded Systems Using LEGO Mindstorms. IEEE Transactions on Education, 52, 99-108. http://dx.doi.org/10.1109/TE.2008.919809
Berges, M. and Hubwieser, P. (2011) Minimally Invasive Programming Courses: Learning OOP with (out) Instruction. Proceedings of the 42nd ACM Technical Symposium on Computer Science Education, Dallas, 9-12 March 2011, 7-92.
Faux, R. (2006) Impact of Preprogramming Course Curriculum on Learning in the First Programming Course. IEEE Transactions on Education, 49, 11-15. http://dx.doi.org/10.1109/TE.2005.852593
Jacobson, M.L., Said, R.A. and Rehman, H. (2006) Introducing Design Skills at the Freshman Level: Structured Design Experience. IEEE Transactions on Education, 49, 247-253. http://dx.doi.org/10.1109/TE.2006.872403
Verginis, I., Gogoulou, A., Gouli, E., Boubouka, M. and Grigoriadou, M. (2001) Enhancing Learning in Introductory Computer Science Courses through SCALE: An Empirical Study. IEEE Transactions on Education, 54, 1-13. http://dx.doi.org/10.1109/TE.2010.2040477
Lahtinen, E. (2007) A Categorization of Novice Programmers: A Cluster Analysis Study. Proceedings of the 19th Annual Workshop of the Psychology of Programming Interest Group, Joensuu, 2-6 July 2007, 32-41.
Vogel-Heuser, B., Obermeier, M., Braun, S., Sommer, K., Jobst, F. and Schweizer, K. (2013) Evaluation of a UML-Based versus an IEC 61131-3-Based Software Engineering Approach for Teaching PLC Programming. IEEE Transactions on Education, 56, 329-335. http://dx.doi.org/10.1109/TE.2012.2226035
International Organization for Standardization (1999) Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs)-Part 11: Guidance on Usability, EN ISO 9241-11:1998. Beuth, Berlin.
Bevan, N. (1995) Measuring Usability as Quality of Use. Software Quality Journal, 4, 115-130. http://dx.doi.org/10.1007/BF00402715
Annett, J. (2003) Hierarchical Task Analysis. In: Hollnagel, E., Ed., Handbook of Cognitive Task Design, Lawrence Erlbaum Assoc. Inc., Mahwah, 17-35.
Lucas, M.R. and Tilbury, D.M. (2002) Quantitative and Qualitative Comparisons of PLC Programs for a Small Testbed with a Focus on Human Issues. Proceedings of the American Control Conference, Anchorage, 5, 4165-4171.
Lucas, M.R. (2003) Understanding and Assessing Logic Control Design Methodologies. Ph.D. Dissertation, Dept. Mechanical Eng., University of Michigan, Ann Arbor.
Friedrich, D. and Vogel-Heuser, B. (2007) Benefit of System Modeling in Automation and Control Education. American Control Conference (ACC), New York, 9-13 July 2007, 2497-2502.
Friedrich, D. (2009) Anwendbarkeit von Methoden und Werkzeugen des konventionellen Softwareengineerings zur Modellierung und Programmierung von Steuerungssystemen. Ph.D. Dissertation, University Kassel, Kassel.
Friedrich, D. and Vogel-Heuser, B. (2005) Evaluating the Benefit of Modeling Notations on the Quality of PLC-Programming. 11th International Conference on Human-Computer Interaction (HCI), Las Vegas, 22-27 July 2005.
Katzke, U. and Vogel-Heuser, B. (2005) UML-PA as an Engineering Model for Distributed Process Automation. IFAC World Conference, Prague, 3-8 July 2005, 129-134.
Katzke, U. and Vogel-Heuser, B. (2009) Vergleich der Anwendbarkeit von UML und UML-PA in der anlagennahen Softwareentwicklung der Automatisierungstechnik-Beispiel einer vergleichenden empirischen Untersuchung von Modellierungssprachen. at-Automatisierungstechnik, 57, 332-340. http://dx.doi.org/10.1524/auto.2009.0781
Witsch, D., Ricken, M., Kormann, B. and Vogel-Heuser, B. (2010) PLC-Statecharts: An Approach to Integrate UML-Statecharts in Open-Loop Control Engineering. 8th IEEE International Conference on Industrial Informatics (INDIN), Osaka, 13-16 July 2010, 915-920.
Witsch, D. and Vogel-Heuser, B. (2009) Close Integration between UML and IEC 61131-3: New Possibilities through Object-Oriented Extensions. IEEE International Conference on Emerging Technologies & Factory Automation (ETFA), Mallorca, 22-26 September 2009, 1-6.
Yang, S. and Sun, J.L. (2010) Modeling Traverse Feature in Concurrent Software System with UML Statecharts. IEEE International Conference on Computational Intelligence and Software Engineering (CiSE), Wuhan, 10-12 December 2010, 133-138.
Witsch, D. (2012) Modellgetriebene Entwicklung von Steuerungssoftware auf Basis der UML unter Berücksichtigung der dom?nenspezifischen Anforderungen des Maschinen-und Anlagenbaus. Ph.D. Dissertation, Faculty of Mechanical Engineering, TUM, Munich.
Vogel-Heuser, B., Seidel, T., Braun, S., Obermeier, M., Sommer, K. and Johannes, C. (2011) Modeling Order Effects on Errors in Object Oriented Modeling for Machine and Plant Automation from an Educational Point of View. 16th IEEE International Conference on Emerging Technologies & Factory Automation (ETFA), Toulouse, 5-9 September 2011.
Frank, T., Eckert, K., Hadlich, T., Fay, A., Diederich, C. and Vogel-Heuser, B. (2012) Workflow and Decision Support for the Design of Distributed Automation Systems. IEEE INDIN: International Conference on Industrial Informatics, Beijing, 25-27 July 2012, 293-299. http://dx.doi.org/10.1109/INDIN.2012.6300859
Frank, T., Hadlich, T., Eckert, K., Fay, A., Diedrich, C. and Vogel-Heuser, B. (2012) Using Contact Points to Integrate Discipline Spanning Real-Time Requirements in Modeling Networked Automation Systems for Manufacturing Systems. IEEE International Conference on Automation Science and Engineering (CASE), Seoul, 20-24 August 2012, 851-856.
Frank, T., Eckert, K., Hadlich, T., Fay, A., Diederich, C. and Vogel-Heuser, B. (2013) Erweiterung des V-Modells? für den Entwurf von verteilten Automatisierungssystemen. At-Automatisierungstechnik, 61, 79-91. http://dx.doi.org/10.1524/auto.2013.0009
Eckert, K., Hadlich, T., Fay, A., Diederich, C. and Vogel-Heuser, B. (2012) Design Patterns for Distributed Automation Systems with Consideration of Non-Functional Requirements. 17th IEEE International Conference on Emerging Technologies & Factory Automation (ETFA), Krakow, 17-21 September 2012, 1-9.
Oswald, W.D. and Roth, E. (1978) Der Zahlen-Verbindungs-Test (ZVT). Ein sprachfreier Intelligenz-Schnell-Test. Verlag für Psychologie Horgrefe, G?ttingen.
Hart, S. and Staveland, L. (1988) Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In: Hancock, P. and Meshkati, N., Eds., Human Mental Workload, North Holland, Amsterdam, 139-183.
Vogel-Heuser, B., Braun, S., Kormann, B. and Friedrich, D. (2011) Implementation and Evaluation of UML as Modeling Notation in Object Oriented Software Engineering for Machine and Plant Automation. 18th World Congress of International Federation of Automation Control (IFAC), Milan, 28 August-2 September 2011, 9151-9157.
Ko, A.J. and Myers, B.A. (2005) A Framework and Methodology for Studying the Causes of Software Errors in Programming Systems. Journal of Visual Languages and Computing, 16, 41-84. http://dx.doi.org/10.1016/j.jvlc.2004.08.003
McCabe, T. (1976) A Complexity Measure. IEEE Transactions on Software Engineering, 2, 308-320. http://dx.doi.org/10.1109/TSE.1976.233837