With the development of the Industrial Internet of Things (IIoT) and cloud computing technologies, intelligent sensors in the field that can generate large volumes of time-series data continuously have emerged. Due to the lack of equipment and network impacts, highly distributed industrial applications cannot capture and transfer all production data to a distant cloud server in real time. Consequently, a portion of critical production data is lost, which poses the significant challenge of the timely replenishment of missing data. Employing deep learning in the cloud center for data trend prediction based on relevant data can solve this problem. The objective of this study was to develop a time-series prediction model that combines a Transformer model with a sparse Mixture of Experts (MoE). The model is designed specifically for an IIoT system that is used in oil-well operations. The proposed TransMoE prediction model combines the advantages of the MoE and the Transformer model. The MoE can effectively handle multiple subtasks while the Transformer algorithm can reflect the long-range dependency of the input data series. The proposed model was used to predict oil-well yields, and the predicted outcomes were compared with those obtained using a CNN-GRU and CNN-LSTM models, as well as the actual recorded data. The experimental results indicated that the proposed TransMoE model can significantly increase the efficiency and accuracy of oil well production sequence data prediction, with an average relative error of 6.26%, which can satisfy the requirements of enterprise data usage.
KeywordsTransformer ModelIndustrial Internet of ThingsMultivariate Time-Series PredictionSelf-Attention MechanismMixture of Experts
Obodoeze, F.C. (2012) Wireless Sensor Network in Niger Delta Oil and Gas Field Monitoring: The Security Challenges and Countermeasures. International Journal of Distributed and Parallel systems , 3, 65-77. https://doi.org/10.5121/ijdps.2012.3606
Shi, F., Yan, L., Zhao, X. and Xian-Ke Gao, R. (2022) Machine Learning-Based Time-Series Data Analysis in Edge-Cloud-Assisted Oil Industrial IoT System. Mobile Information Systems , 2022, Article ID: 5988164. https://doi.org/10.1155/2022/5988164
Al-Shabandar, R., Jaddoa, A., Liatsis, P. and Hussain, A.J. (2021) A Deep Gated Recurrent Neural Network for Petroleum Production Forecasting. Machine Learning with Applications , 3, Article ID: 100013. https://doi.org/10.1016/j.mlwa.2020.100013
Zhang, F., Liu, M., Zhou, Z. and Shen, W. (2016) An Iot-Based Online Monitoring System for Continuous Steel Casting. IEEE Internet of Things Journal , 3, 1355-1363. https://doi.org/10.1109/jiot.2016.2600630
Wanasinghe, T.R., Gosine, R.G., James, L.A., Mann, G.K.I., de Silva, O. and Warrian, P.J. (2020) The Internet of Things in the Oil and Gas Industry: A Systematic Review. IEEE Internet of Things Journal , 7, 8654-8673. https://doi.org/10.1109/jiot.2020.2995617
Aalsalem, M.Y., Khan, W.Z., Gharibi, W. and Armi, N. (2017) An Intelligent Oil and Gas Well Monitoring System Based on Internet of Things. 2017 International Conference on Radar , Antenna , Microwave , Electronics , and Telecommunications ( ICRAMET ), Jakarta, 23-24 October 2017, 124-127. https://doi.org/10.1109/icramet.2017.8253159
Gao, M.J., Xu, J., Tian, J.W. and Zhang, F. (2008) Zigbee Wireless Mesh Networks for Remote Monitoring System of Pumping Unit. 2008 7 th World Congress on Intelligent Control and Automation , Chongqing, 25-27 June 2008, 5901-5905. https://doi.org/10.1109/wcica.2008.4592834
Trappey, A.J.C., Trappey, C.V., Hareesh Govindarajan, U., Chuang, A.C. and Sun, J.J. (2017) A Review of Essential Standards and Patent Landscapes for the Internet of Things: A Key Enabler for Industry 4.0. Advanced Engineering Informatics , 33, 208-229. https://doi.org/10.1016/j.aei.2016.11.007
Gubbi, J., Buyya, R., Marusic, S. and Palaniswami, M. (2013) Internet of Things (IoT): A Vision, Architectural Elements, and Future Directions. Future Generation Computer Systems , 29, 1645-1660. https://doi.org/10.1016/j.future.2013.01.010
Borgia, E. (2014) The Internet of Things Vision: Key Features, Applications and Open Issues. Computer Communications , 54, 1-31. https://doi.org/10.1016/j.comcom.2014.09.008
Wu, Y., Liu, Y., Ahmed, S.H., Peng, J. and Abd El-Latif, A.A. (2020) Dominant Data Set Selection Algorithms for Electricity Consumption Time-Series Data Analysis Based on Affine Transformation. IEEE Internet of Things Journal , 7, 4347-4360. https://doi.org/10.1109/jiot.2019.2946753
Heidari Kapourchali, M. and Banerjee, B. (2018) Unsupervised Feature Learning from Time-Series Data Using Linear Models. IEEE Internet of Things Journal , 5, 3918-3926. https://doi.org/10.1109/jiot.2018.2845340
Akbar, A., Khan, A., Carrez, F. and Moessner, K. (2017) Predictive Analytics for Complex IoT Data Streams. IEEE Internet of Things Journal , 4, 1571-1582. https://doi.org/10.1109/jiot.2017.2712672
Sagheer, A. and Kotb, M. (2019) Time Series Forecasting of Petroleum Production Using Deep LSTM Recurrent Networks. Neurocomputing , 323, 203-213. https://doi.org/10.1016/j.neucom.2018.09.082
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., et al . (2021) Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence , 35, 11106-11115. https://doi.org/10.1609/aaai.v35i12.17325
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., et al . (2021) CvT: Introducing Convolutions to Vision Transformers. 2021 IEEE / CVF International Conference on Computer Vision ( ICCV ), Montreal, 10-17 October 2021, 22-31. https://doi.org/10.1109/iccv48922.2021.00009
Lepikhin, D., Lee, H.J., Xu, Y., et al . (2020) GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv: 2006.16668.
Barret, Z., Bello, Z., Kumar, S., et al . (2022) ST-MoE: Designing Stable and Transferable Sparse Expert Models. arXiv: 2202.08906.
Stassin, S., Corduant, V., Mahmoudi, S.A. and Siebert, X. (2024) Explainability and Evaluation of Vision Transformers: An In-Depth Experimental Study. Electronics , 13, Article 175. https://doi.org/10.3390/electronics13010175
Yan, F., Li, S., Zhou, Z. and Shi, Y. (2024) A Residual Network with Efficient Transformer for Lightweight Image Super-Resolution. Electronics , 13, Article 94. https://doi.org/10.3390/electronics13010194
Minoli, D., Sohraby, K. and Occhiogrosso, B. (2017) IoT Considerations, Requirements, and Architectures for Smart Buildings—Energy Optimization and Next-Generation Building Management Systems. IEEE Internet of Things Journal , 4, 269-283. https://doi.org/10.1109/jiot.2017.2647881
Verma, S., Kawamoto, Y., Fadlullah, Z.M., Nishiyama, H. and Kato, N. (2017) A Survey on Network Methodologies for Real-Time Analytics of Massive IoT Data and Open Research Issues. IEEE Communications Surveys & Tutorials , 19, 1457-1477. https://doi.org/10.1109/comst.2017.2694469
Seren, H.R., Buzi, E., Al-Maghrabi, L., Ham, G., Bernero, G. and Deffenbaugh, M. (2018) An Untethered Sensor Platform for Logging Vertical Wells. IEEE Transactions on Instrumentation and Measurement , 67, 798-803. https://doi.org/10.1109/tim.2017.2774183
Akbar, A., Kousiouris, G., Pervaiz, H., Sancho, J., Ta-Shma, P., Carrez, F., et al . (2018) Real-time Probabilistic Data Fusion for Large-Scale IoT Applications. IEEE Access , 6, 10015-10027. https://doi.org/10.1109/access.2018.2804623
Wang, Y., Shen, Y., Mao, S., Chen, X. and Zou, H. (2019) LASSO and LSTM Integrated Temporal Model for Short-Term Solar Intensity Forecasting. IEEE Internet of Things Journal , 6, 2933-2944. https://doi.org/10.1109/jiot.2018.2877510
Zhang, Y., Thorburn, P.J., Xiang, W. and Fitch, P. (2019) SSIM—A Deep Learning Approach for Recovering Missing Time Series Sensor Data. IEEE Internet of Things Journal , 6, 6618-6628. https://doi.org/10.1109/jiot.2019.2909038
Wang, X., Pi, D., Zhang, X., Liu, H. and Guo, C. (2022) Variational Transformer-Based Anomaly Detection Approach for Multivariate Time Series. Measurement , 191, Article ID: 110791. https://doi.org/10.1016/j.measurement.2022.110791
Morid, M.A., Sheng, O.R.L. and Dunbar, J. (2023) Time Series Prediction Using Deep Learning Methods in Healthcare. ACM Transactions on Management Information Systems , 14, 1-29. https://doi.org/10.1145/3531326
Basha, E., Jurdak, R. and Rus, D. (2014) In-network Distributed Solar Current Prediction. ACM Transactions on Sensor Networks , 11, 1-28. https://doi.org/10.1145/2629593
Piccialli, F., Giampaolo, F., Prezioso, E., Crisci, D. and Cuomo, S. (2021) Predictive Analytics for Smart Parking: A Deep Learning Approach in Forecasting of IoT Data. ACM Transactions on Internet Technology , 21, 1-21. https://doi.org/10.1145/3412842
Chang, X., Li, G., Xing, G., Zhu, K. and Tu, L. (2021) DeepHeart: A Deep Learning Approach for Accurate Heart Rate Estimation from PPG Signals. ACM Transactions on Sensor Networks , 17, 1-18. https://doi.org/10.1145/3441626
Fang, H., Liu, Y., Chen, C. and Hwang, F. (2022) Travel Time Prediction Method Based on Spatial-Feature-Based Hierarchical Clustering and Deep Multi-Input Gated Recurrent Unit. ACM Transactions on Sensor Networks , 19, 1-21. https://doi.org/10.1145/3544976
Vallés-Pérez, I., Soria-Olivas, E., Martínez-Sober, M., Serrano-López, A.J., Gómez-Sanchís, J. and Mateo, F. (2022) Approaching Sales Forecasting Using Recurrent Neural Networks and Transformers. Expert Systems with Applications , 201, Article ID: 116993. https://doi.org/10.1016/j.eswa.2022.116993