Research ArticleOpen AccessGoogle Scholar indexed
Research on Establish an Efficient Log Analysis System with Kafka and Elastic Search
Henan University of Technology, Zhengzhou, China
Henan University of Economics and Law, Zhengzhou, China
Beijing Jiaotong University, Beijing, China
- 1 Henan University of Technology, Zhengzhou, China
- 2 Henan University of Economics and Law, Zhengzhou, China
- 3 Beijing Jiaotong University, Beijing, China
Journal of Software Engineering and Applications·Volume 10 (2017)·Pages 843–853·Published 27 October 2017·DOI10.4236/jsea.2017.1011047
Copy link · social · email
Abstract
Search logs in a timely and efficient manner are an important part of SRE (Site Reliability Engineer). Logs help us solve the problems during our development work. In this paper, we will introduce you a way how to build an efficient logs analysis system based on kafka and Elastic Search. We hope you can learn something through the iteration of the Version and get some inspiration with your own log analysis system.
KeywordsLog Analysis SystemKafkaElastic SearchSoftware ApplicationsSite Reliability Engineer
- Bai, J. and Guo, H.B. (2014) Software Integration Research of Large-Scale Logs Real-Time Search Basedon ElasticSearch. Journal of Jilin Normal University (Natural Science Edition), 2014, No. 2.
- Kreps, J., Narkhede, N. and Rao, J. (2011) Kafka: A Distributed Messaging System for Log Processing. http://notes.stephenholiday.com/Kafka.pdf
- Gormley, C. and Tong, Z. (2015) Elasticsearch: The Definitive Guide. O’Reilly Media, Inc., Sebastopol.
- Bagnasco, S. and Berzano, D. (2015) Monitoring of IaaS and Scientific Applications on the Cloud Using the Elasticsearch Ecosystem. Journal of Physics: Conference Series, 608, No. 1. https://doi.org/10.1088/1742-6596/608/1/012016
- Narkhede, N., Shapira, G. and Palino, T. (2017) Kafka: The Definitive Guide. O’Reilly Media, Inc., Sebastopol.
- Jiang, K., Feng, J., Tang, Z.X. and Wang, C. (2015) The Metadata Searching and Sharing Platform Based on ElasticSearch. Computer and Modernization, No. 2.
- Xu, D.H. (2014) Application Research in Vehicle License Plate Recognition System Based on Elaticsearch. Computer Era, No. 12.
- Lei, X.F., Wang, Z. and He, Y.Z. (2015) Log Real-Time Management Scheme Based on International Workshop on Materials Engineering and Computer Sciences. The International MultiConference of Engineers and Computer Scientists 2015, Hong Kong, 18-20 March 2015.
- Baker, J., et al. (2011) Megastore: Providing Scalable, Highly Available Storage for Interactive Services. Fifth Biennial Conference on Innovative Data Systems Research, Asilomar, January 9-12 2011.
- Nguyen, C.N., Kim, J.-S. and Hwang, S. (2016) KOHA: Building a Kafka-Based Distributed Queue System on the Fly in a Hadoop Cluster. IEEE 1st International Workshops on Foundations and Applications of Self* Systems (FAS*W), Augsburg, 12-16 September 2016, 48-53. https://doi.org/10.1109/FAS-W.2016.23
- Chen, J.J. and Huang, G.F. (2015) Reconstruct Library Search Engine Based on Elasticsearch. Information Research, No. 11.