How to install ELK (Elasticsearch, Logstash, and Kibana)

In this post, we will explain how to install Elasticsearch, Kibana, and Logstash. With these 3 tools, we can centralize our logs, perform advanced searches, or analyze data.

Install Java JDK

To install JDK, we will add the ppa repository, update the repositories, and install java by running:

# add-apt-repository ppa:webupd8team/java
# apt-get update
# apt-get install oracle-java8-installer
# java -version

The last command lets us check whether it was installed correctly. It should display something similar to:

java version “1.8.0_144”
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

Install Elasticsearch

Elasticsearch is a distributed search and analytics engine. To install it, we will add the Elastic repository and proceed with the installation:

# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
# apt-get update
# apt-get install elasticsearch

Now we will change the Elasticsearch configuration to use our server's private IP:

# vi /etc/elasticsearch/elasticsearch.yml

We will add or modify:

network.host: IP-Privada

Now we will start Elasticsearch and configure it to start automatically when the server restarts.

# systemctl start elasticsearch
# systemctl enable elasticsearch

Install Kibana

Kibana lets you explore, visualize, or discover data. We can also generate graphs or place geographic data on any map. To install Kibana:

# apt-get install kibana

We will edit the file vi /etc/kibana/kibana.yml and add or modify:

server.host: "IP-Privada"
elasticsearch.url: "http://IP-Privada:9200"

 Now we will start Kibana and configure it to start automatically when the server restarts.

# systemctl start kibana
# systemctl enable kibana

Install Nginx

Kibana runs on port 5601. We will use Nginx to redirect all traffic from port 80 to port 5601 using the user adminkibana and the password we choose.

# apt-get install nginx apache2-utils
# htpasswd -c /etc/nginx/htpasswd.kibana adminkibana
# echo > /etc/nginx/sites-available/default

We will edit the file /etc/nginx/sites-available/default and modify server_name and proxy_pass:

server {
    listen 80;
 
    server_name IP-Publica;
 
    auth_basic "Acceso a Kibana";
    auth_basic_user_file /etc/nginx/htpasswd.kibana;
 
    location / {
        proxy_pass http://IP-Privada:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

 Now we will start Nginx and configure it to start automatically when the server restarts:

# systemctl start nginx
# systemctl enable nginx

Install Logstash

Logstash lets us centralize, transform, and store our data. In the following article, we show you more examples of filters. To install Logstash, we will run:

# apt-get install logstash

Now we will create the following filters, content for /etc/logstash/conf.d/02-beats-input.conf:

input {
  beats {
    port => 5044
    ssl => false
  }
}

Content for /etc/logstash/conf.d/10-syslog-filter.conf:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
      add_field => [ "program", "%{syslog_program}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

 Now we will modify hosts in /etc/logstash/conf.d/30-elasticsearch-output.conf :

output {
  elasticsearch {
    hosts => "IP-Privada:9200"
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Now we will start Logstash and configure it to start automatically when the server restarts.

# systemctl start logstash
# systemctl enable logstash

Load Beats Dashboards

# wget https://github.com/elastic/beats-dashboards/archive/master.zip
# unzip master.zip
# cd beats-dashboards-master

We will edit load.sh and modify the line containing ELASTICSEARCH:

ELASTICSEARCH=http://IP-Privada:9200

We will access Kibana at http://IP-Pública with the username and password we created in “Nginx installation”. Within Kibana, we will go to Management > Index Patterns, click filebeat-*, and then click the “star”. This sets this filter as the default.

Our ELK is now running:

You may be interested in these other articles: Filters for Logstash or How to send data to ELK using Filebeats.