In our guide today, we are going to look at how to install Elastic stack (ELK) 7.x on Debian 10. ELK is an acronym for Elasticsearch, Logstash and Kibana. ELK is a collection of opensource applications that enables you to collect, analyze and visualize logs from various sources. Each of the components perform specific tasks as explained below:

  • Elasticsearch – search and analytics engine
  • Kibana – data visualization and dash-boarding tool for analyzing data stored on Elasticsearch.
  • Logstash – server‑side data processing pipeline. It ingests data from multiple sources simultaneously and presents to Elasticsearch
  • Beats – Log shippers that collects logs from servers and send them to Logstash or directly to Elasticsearch.

Installing Elastic Stack 7 on Debian 10

The below steps will guide us on how to get Elastic Stack 7 installed on Debian 10. Note that I am running the commands as root user.

Step 1: Update System Packages

Update your system packages to begin your installation

sudo apt-get update
sudo apt-get upgrade
sudo reboot

Step 2: Install Java on Debian 10

ELK deployment requires that Java 8 or 11 is installed. Run the below commands to install OpenJDK 11

sudo apt install openjdk-11-jdk -y

Confirm Java installation by checking on the version

$ java --version
openjdk 11.0.9.1 2020-11-04 
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Debian-1deb10u2) 
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-post-Debian-1deb10u2, mixed mode, sharing)

Step 3: Add Elastic Stack Repository to Debian 10

Install Elastic stack PGP signing key with the below command:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Install ELK APT repository on Debian 10

sudo apt install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

Update package list cache:

sudo apt update

Step 4: Install Elasticsearch on Debian 10

Once we have successfully added ELK repo,we can go ahead to install the different components of elactic stack. To install elasticsearch, run the below command:

sudo apt install elasticsearch

Now we need to configure Elasticsearch to define the IP address and the port to listen on. Also set discovery type and cluster name. The configuration file is found in /etc/elasticsearch/elasticsearch.yml. Change the settings as below:

# ---------------------------------- Cluster ----------------------------------- 
... 
cluster.name: debian-10

# ---------------------------------- Network ----------------------------------- 
... 
network.host: 192.168.50.2 
... 
http.port: 9200

# --------------------------------- Discovery ---------------------------------- 
... 
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: []
---
discovery.type: single-node

Also set JVM heap size to about the available memory on your system.

$ sudo vim /etc/elasticsearch/jvm.options 
# Xms represents the initial size of total heap space 
# Xmx represents the maximum size of total heap space 

-Xms512m 
-Xmx512m

Save the file then start and enable Elasticsearch as below:

sudo systemctl enable --now elasticsearch

Confirm status with the following command:

$ systemctl status elasticsearch 
● elasticsearch.service - Elasticsearch 
  Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) 
  Active: active (running) since Mon 2021-01-18 11:49:06 EAT; 1min 1s ago 
    Docs: https://www.elastic.co 
Main PID: 2524 (java) 
   Tasks: 49 (limit: 2320) 
  Memory: 1003.9M

Step 5: Install Kibana on Debian 10

Once Elasticsearch is up and running, install Kibana with the below command:

sudo apt install kibana

The default Kibana configuration file is in /etc/kibana/kibana.yml. Configure IP address and port as below

$ sudo vim /etc/kibana/kibana.yml.
# Kibana is served by a back end server. This setting specifies the port to use. 
# server.port: 5601 
server.port: 5601 
... 
# To allow connections from remote users, set this parameter to a non-loopback address. 
#server.host: "localhost" 
server.host: "192.168.50.2"

Use the below settings to configure how Kibana connects to Elasticsearch

# The URLs of the Elasticsearch instances to use for all your queries. 
#elasticsearch.hosts: ["http://localhost:9200"] 
elasticsearch.hosts: ["http://192.168.50.2:9200"]

Enable and start Kibana

sudo systemctl enable --now kibana

Confirm Kibana status

$ systemctl status kibana
● kibana.service - Kibana 
  Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) 
  Active: active (running) since Mon 2021-01-18 12:46:08 EAT; 32s ago 
Main PID: 4359 (node) 
   Tasks: 7 (limit: 2320) 
  Memory: 49.8M 
  CGroup: /system.slice/kibana.service 
          └─4359 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist

Configure firewall to allow Kibana port for Kibana to be accessible from the internet

sudo ufw allow 5601/tcp

Access Kibana dashboard from the browser using your server IP or hostname and Kibana port 5601: http://<server-ip-address>:5601

Elastic Stack 7

Step 6: Install Logstash on Debian 10

Once Kibana is running, run the below command to install Logstash

sudo apt-get install logstash

Configure Logstash

Create a configuration file named 02-beats-input.conf where you will set up your Filebeat input

sudo vim  /etc/logstash/conf.d/02-beats-input.conf

Add the following content

input { 
 beats { 
   port => 5044 
 } 
}

Create another configuration file to add filters configurations for system logs

sudo vim /etc/logstash/conf.d/10-syslog-filter.conf

Add the following content to the file. This is just an example of a configuration for parsing incoming system logs to make them structured and usable by the Kibana dashboards:

filter {
      if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
        }
        syslog_pri { }
        date {
          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
      }
    }

Lastly, create another configuration file where we are going to tell Logstash to store Beats data in Elasticsearch.

sudo vim /etc/logstash/conf.d/30-elasticsearch-output.conf

Put the following content

output { 
 elasticsearch { 
   hosts => ["192.168.50.2:9200"] 
   manage_template => false 
   index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
 } 
}

Start and enable Logstash with the below commands:

sudo systemctl start logstash
sudo systemctl enable logstash

Step 7: Install Filebeat on Debian 10

As explained earlier, ELK uses beats to ship data from various sources and present them to either Logstash or Elasticsearch. Below are some of the beats and what they do:

  • Filebeat: collects and ships log files.
  • Metricbeat: collects metrics from your systems and services.
  • Packetbeat: collects and analyzes network data.
  • Winlogbeat: collects Windows event logs.
  • Auditbeat: collects Linux audit framework data and monitors file integrity.
  • Heartbeat: monitors services for their availability with active probing.

In this tutorial we are going to install filebeat on the same server as Elasticsearch

sudo apt-get install filebeat

Now configure filebeat to send data to Logstash:

sudo vim /etc/filebeat/filebeat.yml

In the output section, comment out Elasticsearch and enable Logstash output

#-------------------------- Elasticsearch output ------------------------------ 
# output.elasticsearch: 
 # Array of hosts to connect to. 
 # hosts: ["localhost:9200"] 

 # Optional protocol and basic auth credentials. 
 #protocol: "https" 
 #username: "elastic" 
 #password: "changeme" 

#----------------------------- Logstash output -------------------------------- 
output.logstash: 
 # The Logstash hosts 
 hosts: ["192.168.50.2:5044"] 
...

Enable Filebeat Modules

The modules collect and parse system logs. Enable as below:

sudo filebeat modules enable system

Load Index Template

We need to load the template to elasticsearch manually since we have configured our output to Logstash. Run the command below:

sudo filebeat setup \
  --index-management -E output.logstash.enabled=false \
  -E 'output.elasticsearch.hosts=["192.168.50.2:9200"]'

Now start and enable Filebeat as below:

sudo systemctl start filebeat
sudo systemctl enable filebeat

Next is to login to Kibana to add Filebeat index. Under ‘Management‘, click on ‘Stack Management‘ then under ‘Kibana‘, click on ‘Index Patterns‘. Click on ‘Create Index Pattern‘ to add a new index

In the index pattern name, enter ‘filebeat‘ and the system will search for matching patterns. It found filebeat on my case

Click ‘Next Step’ and configure timestamp

Then click on ‘Create Index Pattern’. Your new index will be configured. Go back to the left pane menu, under ‘Kibana’, click ‘Discover‘ and your data should be imported.

That’s it. You have successfully installed and configured Elastic stack on Debian 10. Note that the order of installation should be elasticsearch, kibana, logstash and finally beats. When beats are installed on remote servers, ensure that they have connectivity to the elastic stack server. I hope the guide has been useful. Check more interesting guides below:

LEAVE A REPLY

Please enter your comment!
Please enter your name here