Reinforcement Learning in Mechatronic Systems: A Case Study on DC Motor Control — Oak Academic Publishing
Research ArticleOpen AccessGoogle Scholar indexed
Reinforcement Learning in Mechatronic Systems: A Case Study on DC Motor Control
CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
,
Heringer Consulting GmbH, Cologne, Germany
,
CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
,
CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
,
CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
,
CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
,
Division of Electricity, Department of Electrical Engineering, Uppsala University, Uppsala, Sweden
,
CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
1 CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
2 Heringer Consulting GmbH, Cologne, Germany
3 CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
4 CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
5 CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
6 CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
7 Division of Electricity, Department of Electrical Engineering, Uppsala University, Uppsala, Sweden
8 CAD CAM Center Cologne, Institute of Automotive Engineering Cologne (IFK), Faculty of Automotive Systems and Production, Cologne University of Applied Science, Cologne, Germany
The integration of artificial intelligence into the development and production of mechatronic products offers a substantial opportunity to enhance efficiency, adaptability, and system performance. This paper examines the utilization of reinforcement learning as a control strategy, with a particular focus on its deployment in pivotal stages of the product development lifecycle, specifically between system architecture and system integration and verification. A controller based on reinforcement learning was developed and evaluated in comparison to traditional proportional-integral controllers in dynamic and fault-prone environments. The results illustrate the superior adaptability, stability, and optimization potential of the reinforcement learning approach, particularly in addressing dynamic disturbances and ensuring robust performance. The study illustrates how reinforcement learning can facilitate the transition from conceptual design to implementation by automating optimization processes, enabling interface automation, and enhancing system-level testing. Based on the aforementioned findings, this paper presents future directions for research, which include the integration of domain-specific knowledge into the reinforcement learning process and the validation of this process in real-world environments. The results underscore the potential of artificial intelligence-driven methodologies to revolutionize the design and deployment of intelligent mechatronic systems.
KeywordsArtificial Intelligence in Product DevelopmentMechatronic SystemsReinforcement Learning for ControlSystem Integration and VerificationAdaptive Optimization ProcessesKnowledge-Based Engineering
Åström, K.J. and Hägglund, T. (2006) Advanced PID Control. ISA—The Instrumentation, Systems, and Automation Society.
Lloyds Raja, G. and Ali, A. (2021) New PI-PD Controller Design Strategy for Industrial Unstable and Integrating Processes with Dead Time and Inverse Response. Journal of Control , Automation and Electrical Systems , 32, 266-280. https://doi.org/10.1007/s40313-020-00679-5
Jiménez, G.A., de la Escalera Hueso, A. and Gómez-Silva, M.J. (2023) Reinforcement Learning Algorithms for Autonomous Mission Accomplishment by Unmanned Aerial Vehicles: A Comparative View with DQN, SARSA and A2C. Sensors , 23, Article 9013. https://doi.org/10.3390/s23219013
De La Fuente, N. and Guerra, D.A.V. (2024) A Comparative Study of Deep Reinforce-ment Learning Models: DQN vs PPO vs A2C. arXiv: 2407.14151.
Vouros, G.A. (2023) Explainable Deep Reinforcement Learning: State of the Art and Challenges. arXiv: 2301.09937.
Sutton, R.S. and Barto, A. (2020) Reinforcement Learning: An Introduction. Second Edition, The MIT Press.
Gräßler, I. (2021) VDI/VDE 2206: Entwicklung mechatronischer und cyber-physischer Systeme. Inhaltsverzeichnis.
Gatti, C. (2015) Design of Experiments for Reinforcement Learning. Springer.
Ladosz, P., Weng, L., Kim, M. and Oh, H. (2022) Exploration in Deep Reinforcement Learning: A Survey. Information Fusion , 85, 1-22. https://doi.org/10.1016/j.inffus.2022.03.003
Fazdi, M.F. and Hsueh, P. (2023) Parameters Identification of a Permanent Magnet DC Motor: A Review. Electronics , 12, Article 2559. https://doi.org/10.3390/electronics12122559
Shah, R. and Sands, T. (2021) Comparing Methods of DC Motor Control for UUVs. Applied Sciences , 11, Article 4972. https://doi.org/10.3390/app11114972
Aribowo, W., Supari, S. and Suprianto, B. (2022) Optimization of PID Parameters for Controlling DC Motor Based on the Aquila Optimizer Algorithm. International Journal of Power Electronics and Drive Systems ( IJPEDS ), 13, 216-222. https://doi.org/10.11591/ijpeds.v13.i1.pp216-222
Spring, E. (2009) Elektrische Maschinen: Eine Einführung. Springer.
Borase, R.P., Maghade, D.K., Sondkar, S.Y. and Pawar, S.N. (2020) A Review of PID Control, Tuning Methods and Applications. International Journal of Dynamics and Control , 9, 818-827. https://doi.org/10.1007/s40435-020-00665-4
Chau, P.C. (2002) Process Control. Cambridge University Press. https://doi.org/10.1017/cbo9780511813665
Zhang, A., McAllister, R., Calandra, R., Gal, Y. and Levine, S. (2020) Learning Invariant Representations for Reinforcement Learning without Reconstruction. arXiv: 2006.10742.
Schwarzer, M., Anand, A., Goel, R., Hjelm, R.D., Courville, A. and Bachman, P. (2020) Data-Efficient Reinforcement Learning with Self-Predictive Representations. arXiv: 2007.05929.
Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., et al . (2020) Scipy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17, 261-272. https://doi.org/10.1038/s41592-019-0686-2
Pinto, V., Gonçalves, J. and Costa, P. (2020) Modeling and Control of a DC Motor Coupled to a Non-Rigid Joint. Applied System Innovation , 3, Article 24. https://doi.org/10.3390/asi3020024
Wang, Y. and Shao, H. (2000) Optimal Tuning for PI Controller. Automatica , 36, 147-152. https://doi.org/10.1016/s0005-1098(99)00130-2
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., et al . (2024) Deep Reinforcement Learning: A Survey. IEEE Transactions on Neural Networks and Learning Systems , 35, 5064-5078. https://doi.org/10.1109/tnnls.2022.3207346
Matsuo, Y., LeCun, Y., Sahani, M., Precup, D., Silver, D., Sugiyama, M., et al . (2022) Deep Learning, Reinforcement Learning, and World Models. Neural Networks , 152, 267-275. https://doi.org/10.1016/j.neunet.2022.03.037
Arulkumaran, K., Deisenroth, M.P., Brundage, M. and Bharath, A.A. (2017) Deep Reinforcement Learning: A Brief Survey. IEEE Signal Processing Magazine , 34, 26-38. https://doi.org/10.1109/msp.2017.2743240
Haarnoja, T., Zhou, A., Abbeel, P. and Levine, S. (2018) Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv: 1801.01290.
Zhang, K., Wang, Z., Chen, G., Zhang, L., Yang, Y., Yao, C., et al . (2022) Training Effective Deep Reinforcement Learning Agents for Real-Time Life-Cycle Production Optimization. Journal of Petroleum Science and Engineering , 208, Article ID: 109766. https://doi.org/10.1016/j.petrol.2021.109766
Laskin, M., et al . (2021) URLB: Unsupervised Reinforcement Learning Benchmark. arXiv: 2110.15191.
Eschmann, J. (2021) Reward Function Design in Reinforcement Learning. In: Belousov, B., Abdulsamad, H., Klink, P., Parisi, S. and Peters, J., Eds., Reinforcement Learning Algorithms : Analysis and Applications , Springer, 25-33. https://doi.org/10.1007/978-3-030-41188-6_3
Kwon, M., Xie, S.M., Bullard, K. and Sadigh, D. (2023) Reward Design with Language Models. arXiv: 2303.00001.
Kiran, M. and Ozyildirim, M. (2022) Hyperparameter Tuning for Deep Reinforcement Learning Applications. arXiv: 2201.11182.
Felten, F., Gareev, D., Talbi, E.G. and Danoy, G. (2023) Hyperparameter Optimization for Multi-Objective Reinforcement Learning. arXiv: 2310.16487.
Dean, A., Voss, D. and Draguljić, D. (2017) Design and Analysis of Experiments. Springer.
Nüssgen, A., et al . (2023) Intelligent Component Manufacturability Testing in Virtual Product Development. Artificial Intelligence und Machine Learning in der CAE - B a sierten Simulation , München, 23-24 Oktober 2023, 14-22.