Header image

Installation of Elasticsearch, Logstash, and Kibana (ELK Stack) With Xpack Security on Ubuntu 20.04

July 15, 2020Devops , ELK

Step 1: Installation of Java

apt-get install openjdk-11-jdk -y

Step 2: Other dependencies

apt-get install nginx curl wget -y

Step 3: Add the Elastic repository

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list

Step 4: Update and install Elasticsearch

apt-get update -y
apt-get install elasticsearch -y


Step 5: Making elasticsearch public

nano /etc/elasticsearch/elasticsearch.yml

add this in config file

transport.host: localhost 
transport.tcp.port: 9300 
http.port: 9200
network.host: 0.0.0.0

Don’t worry we will protect it later

Step 6: Start and enable elasticsearch on reboot

systemctl start elasticsearch

systemctl enable elasticsearch

Step 7: Install kibana, start & enable

apt-get install kibana -y

systemctl start kibana
systemctl enable kibana

Step 8: kibana nginx proxy setup

Create an Nginx virtual host configuration file

nano /etc/nginx/sites-available/kibana

Add the following …Don’t forget to replace example.com

server {
    listen 80;

    server_name kibana.example.com;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Copy same config to sites-enabled

ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/

Restart nginx

systemctl restart nginx

Step 9: Install and Configure Logstash

apt-get install logstash -y

Once Logstash has been installed, create a new beat configuration file with the following command:

nano /etc/logstash/conf.d/02-beats-input.conf

Add the following lines:

input {
  beats {
    port => 5044
  }
}

Save and close the file then create an Elasticsearch configuration file with the following command:

nano /etc/logstash/conf.d/30-elasticsearch-output.conf

Add the following lines:

output {
  if [@metadata][pipeline] {
    elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    pipeline => "%{[@metadata][pipeline]}"
    }
  } else {
    elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

start the Logstash service and enable it to start at boot

systemctl start logstash
systemctl enable logstash

Step 10: Install and Configure Filebeat

ELK stack uses Filebeat to collect data from various sources and transport them to Logstash.

You can install Filebeat with the following command:

apt-get install filebeat -y

Once installed, you will need to configure Filebeat to connect to Logstash. You can configure it with the following command:

nano /etc/filebeat/filebeat.yml

Comment out the following lines:

#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

Then, uncomment the following lines:

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Save and close the file then enable the system module with the following command:

filebeat modules enable system

By default, Filebeat is configured to use default paths for the syslog and authorization logs.

You can load the ingest pipeline for the system module with the following command:

filebeat setup --pipelines --modules system

Next, load the template with the following command:

filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

You should see the following output:

Index setup finished.

By default, Filebeat comes packaged with sample Kibana dashboards that allow you to visualize Filebeat data in Kibana. So you need to disable the Logstash output and enable Elasticsearch output. You can do it with the following command:

filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601

You should see the following output:

Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-ml.html
Loaded machine learning job configurations
Loaded Ingest pipelines

Now, start the Filebeat service and enable it to start at boot with the following command:

systemctl start filebeat
systemctl enable filebeat

Step 11: Securing Elasticsearch with xpack

Open the following:

nano /etc/elasticsearch/elasticsearch.yml

and add this config

xpack.security.enabled: true

then restart the service

systemctl restart elasticsearch

Set passwords

/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

Step 12: Set above password in kibana config

open kibana config file

nano /etc/kibana/kibana.yml

Update and uncomment the following

elasticsearch.username: "elastic"
elasticsearch.password: "Your password"

Restart kibana service

systemctl restart kibana.service

Related post

Monitoring Services with Elasticsearch APM

July 16, 2020Devops , ELK

Step 1: Download and unpack APM Server Download and unpack APM Server Step 2: Edit the configuration (If xpack is enabled) Open config file Change localhost:8200 with 0.0.0.0 Replace host and add username and password of elasticsearch Start and enable

Read more

Leave A Comment

Your email is safe with us.