Enhancing report aggregation and analysis systems with elk platform

ENHANCING REPORT AGGREGATION AND ANALYSIS

SYSTEMS WITH ELK PLATFORM

SVTH: Nhâ m Gia Hoàng Anh

GVHD: ThS Vũ Minh Tuấn

TÓ M TẮT - Dữ liệu là thành phần cơ bản và thiết yếu để xâ y dựng hệ thống phần mềm. Tất cả

cá c hệ thống đều xoay quanh việc tương tác, chuyển đổi và truyền dữ liệu, từ đó tạo nên giá trị cốt lõ i

của doanh nghiệp. Do đó, quản lý dữ liệu sẽ mang lại hiệu quả, lợi í ch và thá ch thức to lớn trong mọi

giai đoạn của hệ thống. Trong bài viết này, những thá ch thức và vấn đề tác động đến doanh nghiệp

sẽ được xem xét cũng như giải phá p tuyệt vời để xử lý chúng, đó chính là ELK. ELK là tên của một

nhó m phần mềm mã nguồn mở, bao gồm Elasticsearch, Logstash và Kibana. Trong đó, Elasticsearch

là nguồn gốc hay trá i tim của hệ thống này. Bài bá o sẽ trì nh bày lý thuyết và ứng dụng thực tế của

ELK để giải quyết vấn đề hội tụ cá c má y chủ của cơ sở dữ liệu phân tán để chứng minh khả năng của

nó .

ABSTRACT - Data is the fundamental and essential component to construct the software

system. All systems revolve around interacting, transforming and transferring data, thereby creating

the core values of businesses. Therefore, data management will bring about great effects, benefits

and challenges in every stage of the system. In this article, challenges and problems that impact

businesses will be considered as well as a great solution to handle them, which is ELK. ELK is the

name of an open source software group, which includes Elasticsearch, Logstash and Kibana. In

which, Elasticsearch is the source or heart of this system. In the deeper part of the article, ELK will be

thoroughly explored, thereby helping readers understand why ELK is a great solution to practical

problems related to data management. The article will cover the theory and practical application of

ELK to solve the problem of converging distributed database’s servers to prove its capabilities.

I. Introduction

The article will discuss the Liberico group's current data problems, as well as how to

identify and incorporate a suitable solution. The ELK stack, an ecosystem of three open

source applications with outstanding data processing capability, represents this approach

course. The simulation of applying ELK stack technology to a company, as well as the results

obtained, will be used to assess ELK's success. This paper serves as a proof of concept.

Currently, software technology is being applied in all forms of business, which brings

great value. However, technological and, more specifically, data risks can occur at any time.

Good data governance is a top priority as well as a significant task for the engineering staff.

During the operation and development of a software system, there will be many risks and

difficulties related to data such as retrieving old data, new data from other software that does

not integrate with the system, etc. Some companies, in particular, have been through many

249

phases of software model transformation, resulting in changes to data systems and

infrastructure. However, depending on the business, they need to recover old data in order to

generate reports and revenue statistics.

This is what happened in the Liberico Group when one of their subsidiaries, BTCOMs -

a business specialized in the sale of retail products and household goods - need to consolidate

the sales reports for the previous year. However, during that one year period BTCOM had a

major change in technology model that resulted in the old data no longer existing in the

current model. To accomplish this task, the responsibility rests on one of the group's other

subsidiaries. IPTECH is a software engineering firm that focuses on delivering applications to

other companies in the enterprise.

This paper’s aim is to concentrate on exploring and developing technical solutions to

help companies overcome the problem mentioned above. The ELK stack is the approach, a

feature-rich and ideal environment for centralizing discrete data sources. This technological

basis can be thoroughly researched and extended to the IPTECH business corporation,

assisting them in dealing with existing and potential challenges in the future. Data

aggregation from different channels is made simpler with the ELK stack.

II. Background Knowledge:

1. ELK Stack:

ELK is a very rich ecosystem that offers a full range of features such as quick search,

fulltext search, analytics stack and a practical interface. ELK consists of 3 open source

projects: Elasticsearch, Logstash and Kibana. In which, Elasticsearch plays the central role of

storing, searching and analyzing data. Logstash is a system for collecting, transporting and

processing data from sources to Elasticsearch. Kibana provides an intuitive ELK stack

interface for users. In which, Elasticsearch is the most important component and is considered

as the heart of this system.

2. Components of ELK stack:

a. Elasticsearch:

Elasticseach is the core concepts behind ELK stack, on this part, Elasticseach will be

defined deeply for the purpose of covering the stack inside-out, starting from heat and moving

on to its surrounding components. Initially released in 2010, Elasticsearch (sometimes dubbed

ES) is a modern search and analytics engine which is based on Apache Lucene (Gedalyah,

2020). Different from traditional relational databases, Elasticsearch use document-oriented to

store the data in an unstructured way. These data, mainly JSON types, are considered as the

first-class citizens in Elasticsearch. They are known as documents, and they are organized in

different indexes and types.

250

Elasticsearch is a Java-based, distributed, open-source search and analytics engine

based on Apache Lucene. It began as a scalable version of the open-source Lucene search

framework, then added the ability to scale Lucene indices horizontally. Elasticsearch allows

user to rapidly store, search, and evaluate large amounts of data in near real-time, with results

arriving in milliseconds. It is able to obtain fast search results because it scans an index rather

than searching the text directly. It has a document-based structure rather than tables and

schemas, and it has comprehensive REST APIs for storing and searching data. Elasticsearch

can be thought of as a server that can process JSON requests and return JSON data at its core.

b. Logstash:

Logstash is built to enrich the ELK ecosystem with crucial missions: Collecting and

transfer data through the data pipelines. (A data pipeline can be easily built with the helps of

Logstash, moreover, not an only engine to transport the data, Logstash can enrich the data

with variety configurations.) Logstash provides user with various ways of collecting data due

to its wide range of input sources. Hence, user can enrich those data with numerous amount of

plugins. here are over 200 different plugins for Logstash, with a vast community making use

of its extensible features (Dotan, 2020). Logstash also play a pivital role of collecting log

messages, which is very useful in tracking the data progress and maintain system. With the

raise of Beats, Logstash usually paired with Beats to become a great data ingestion tool in

ELK’s arsenal.

c. Kibana:

Kibana is the ‘K’ in the ELK Stack, the world’s most popular open source log analysis

platform, and provides users with a tool for exploring, visualizing, and building dashboards

on top of the log data stored in Elasticsearch clusters (Asaf, 2020). User have the chance to

explore the large volume of data due to its simple, friendly interface. Moreover, Kibana

provides such useful tools like searching, analyze and visualize for the deep understanding of

data in realtime. People can utilize its services in creating tables, charts for the advanced

analysis in data. In the Elastic Stack ecosystem, Kibana is the managing and monitoring

officer who consult the best visions for the customers. Developed in 2013 from within the

Elasticsearch community, Kibana has grown to become the window into the Elastic Stack

itself, offering a portal for users and companies

III. Proposed scenario:

1. Scenario for an enterprise e-commerce system

Liberico Group incorporation specializes in the distribution and supply of consumer

products and household appliances at retail locations throughout the country. Currently, the

corporation is made up of around seven independent firms, each of which offers a different

251

form of product. In which, IPTECH is a member of the association tasked with delivering,

maintaining, and updating software systems for organizations under the same enterprise.

IPTECH, on the other hand, just recently joined the group in September 2020. However,

since the beginning of 2019, IPTECH has been a business partner with BTCOM - a subsidiary

of Liberico that specializes in supplying household and consumer goods to rural areas in

Vietnam. From the beginning of its collaboration with BTCOM until its merger with the

Liberico group, IPTECH renovated and developed three version of e-commerce websites and

management systems for BTCOM. In addition, the company also contributes to overcoming

many technological difficulties, providing many technical solutions to stakeholders. 3 e-

commerce and management systems that IPTECH provides to BTCOM are called v1, v2 and

v3 respectively. During the operation and development of these systems, IPTECH has

encountered certain difficulties in data.

2. Data constraints:

At the end of 2020, BTCOM must print invoices and updates on previous year's

merchandise revenues (2019). However, the data from 2019 is no longer in the new system

(v3), but rather in the previous system (v1). Due to the fast and pressing timeline, IPTECH

was forced to fund BTCOM entirely by hand. That is, reboot the v1 system, directly retrieve

the data, and then push the data to the main database of the v3 system. Employees of BTCOM

will then easily retrieve the appropriate data and carry out their tasks. This job requires a lot

of time and meticulousness of members from IPTECH, because the reports and revenue span

many months in 2019, so the amount of data is very large. The reports and revenue recovered

from v1 are still excel files in aggregate form. Therefore, the software developers have to

extract the necessary columns of data and create new tables in the database to be able to

integrate with the v3 system. After all, re-collecting the data necessitates a huge amount of

time and accuracy, since even a minor error will have a significant effect on the group's

consolidated report.

Another issue that arises is system internal stagnation during the re-collection of old

records. Since BTCOM's server would only need to supply services for two major systems, v3

and v2, in regular times. During the aforementioned event, the server was forced to run three

systems: v1, v2, and v3. This has a major impact on the working processes of both BTCOM

and IPTECH participants. Due to unreliable operating systems, the workload increased,

causing significant harm to the BTCOM group.

252

Figure 1: BTCOM’s 3 systems at once

3. Solution:

IPTECH's aim, as can be said, is to upgrade v3 with old data from v1 so that BTCOM

can complete their task. It is not possible to manually obtain data from v1 as previously

stated; instead, a link to v1's database may be used to obtain the old data sources directly.

However, since v3 is a radically new implementation of v1, the information structure and

technologies used in v3 vary significantly from those used in v1. So, if v3 will bind to v1's

database, v3 would have to make a lot of changes. The problem stems from the fact that the

databases are still fragmented and not centralized. Therefore, it is necessary to have a way to

unify these databases into a cluster. However, such homogenization needs a clear strategy to

secure current and long-term goals. Building a perfect model to integrate existing and future

databases will require a large amount of resources, money and time to implement and operate.

There is a less expensive and more efficient method that is using ELK stack. Because ELK

consists of 3 open-source products with powerful capabilities in data integration.

Figure 2: ELK as architecture solution

253

IV. Implementation:

The study has had many anticipated achievements during the development of the ELK

stack model. In which three major issues are addressed:

• Elasticsearch cluster was successfully developed to ensure high data source stability

and scalability.

• Integrate data sources from v1 and v3 'simulated' databases into the ELK stack.

• Query and analyze data fed into the ELK stack through RESTful APIs using the

Elasticsearch and Kibana platforms.

Figure 3: Example of v1’s data inside Kibana platform

V. Conclusion

This paper discussed the data-related issues that businesses face, especially in the

Liberico enterprise. The database infrastructure has been fractured as a result of network

engineering shifts in subsidiaries, causing these issues. And the approach is to integrate the

ELK stack application architecture into the enterprise's computing infrastructure to address

the challenge. The aim of this paper is to demonstrate that the ELK stack can be used to

aggregate data and review reports on Elasticsearch and Kibana platforms, Hence, the paper

254

serves as proof of concept. The deployment process yielded several promising outcomes,

demonstrating the ELK stack's current and future promise.

Aside from the accomplishments, the article must overcome the shortcomings and

provide additional solutions on the ELK stack platform in the future. Limitations, such as the

use of multiple datapiplines in logstash, can result in overburdened servers, necessitating

close consideration of the most optimal datapiplines for the enterprise. Since the program

running the operation is not a true server, certain elasticsearch capabilities have not been fully

used.

More study and operating processes will be needed in the future for an organization to

really use the ELK stack structure. However, the article has partially shown the efficacy of the

ELK architecture, which is the first significant step in connecting the ELK stack to business

applications.

REFERENCES

Gedalyah, R., Daniel, B., (Apr 5, 2020), An Elasticsearch Tutorial: Getting Started.

https://logz.io/blog/elasticsearch-tutorial/

Dotan Horovits, (Jun9, 2020), The Complete Guide To The ELK Stack.

https://logz.io/learn/complete-guide-elk-stack

Asaf Yigal, (Apr 7, 2020), Grafana vs. Kibana: The Key Differences to Know.

https://logz.io/blog/grafana-vs-kibana/

255