Analyzing Nginx logs with Filebeat and Elasticsearch

This article documents the use of Filebeat, Kibana, and Elasticsearch to build a system for collecting and analyzing Nginx logs; Filebeat is responsible for delivering Nginx log data as a data source to Elasticsearch.

As an introduction, we will first explain the relationship between the software: Elasticsearch is a distributed full-text search and data analysis engine based on RESTful style; Kibana is a web system for visualizing Elasticsearch data; and Filebeat is responsible for delivering Nginx log data to Elasticsearch as a data source.

The server environment for this article: Ubuntu 20.04, with only one server node configured.

Environment preparation: JAVA

You need to install Java Runtime Environment (JRE) and Java Development Kit (JDK).

sudo apt update
apt install default-jre
java -version

sudo apt install default-jdk
javac -version

Reference ：How To Install Java with Apt on Ubuntu 20.04

Elasticsearch

Install Elasticsearch

#将Elasticsearch公共GPG密钥导入APT：
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

#添加Elastic源（笔者试验时elasticsearch最新是7.xx版本）：
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

#安装elasticsearch
sudo apt update
sudo apt install elasticsearch

Configuring Elasticsearch

The configuration file for Elasticsearch is located at:

sudo vim /etc/elasticsearch/elasticsearch.yml

#重要配置项：
#集群名称，相同网络下相同的集群名的节点为一个集群。
cluster.name

#存放数据的目录
path.data

#主机地址
network.host: localhost

#服务端口号
http.port: 9200

Managing Elasticsearch.

#启动/停止/重启
sudo systemctl start/stop/restart elasticsearch.service
#注册开机自启动
sudo systemctl enable elasticsearch.service

Verify that the Elasticsearch service is up

root@easeapi:~# curl -X GET localhost:9200
{
  "name" : "easeapi",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "easeapi-wDpaWmxXHJ8yu7a",
  "version" : {
    "number" : "7.11.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "c4ca4238a0b923820dcc509a6f75849b",
    "build_date" : "2021-03-01T08:54:38.141101Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Modifying Elasticsearch memory size

Elasticsearch 7.x is configured differently than before and requires a new file with the .options suffix in the /etc/elasticsearch/jvm.options.d directory. Modify the memory usage configuration as follows.

1
2
3

vim jvm.options.d/jvm.options
-Xms1g
-Xmx1g

Reference: elasticsearch jvm-options

Kibana

The source was added in the previous step, so you can install it directly. Because of version dependencies, install Elasticsearch first before installing Kibana.

`1`	`sudo apt install kibana`

The Elasticsearch configuration file is located at

`1`	`vim /etc/kibana/kibana.yml`

Important configuration items.

server.port: 5601
server.host: "localhost"
server.basePath: "/kibana"//保持和nginx的访问目录配置一致
server.rewriteBasePath: true
elasticsearch.hosts: ["http://localhost:9200"]

#配置为中文
i18n.locale: "zh-CN"

Manage Kibana

#启动/停止/重新启动
sudo systemctl start/stop/restart kibana
#加入自启动
sudo systemctl enable kibana

Filebeat

Filebeat is a lightweight data collection and processing tool with the advantage of low resource consumption.

sudo apt install filebeat
#配置文件
sudo vim /etc/filebeat/filebeat.yml

#主要配置项
setup.kibana:
    host: "localhost:5601"

output.elasticsearch:
    hosts: ["localhost:9200"]

Manage Filebeat

#启动/停止/重新启动
sudo systemctl start/stop/restart filebeat
#加入自启动
sudo systemctl enable filebeat
#测试连通性
filebeat test output

Filebeat is extended by Module to support files with different data formats. module-related commands.

#查看模块状态
sudo filebeat modules list
#启动指定Module
sudo filebeat modules enable nginx

Take Nginx for example, after starting Nginx’s Module, you can see that the file nginx.yml..disabled in the /etc/filebeat/modules.d directory has become nginx.yml.

vim /etc/filebeat/modules.d/nginx.yml

#配置nginx的日志路径。
- module: nginx
    access:
        enabled: true
        var.paths: ["/var/log/nginx/access.log"]

    error:
        enabled: true
        ar.paths: ["/var/log/nginx/error.log"]

Once the configuration is complete, use filebeat setup to write the configuration and then start filebeat.

Removing redundant fields

Filebeat’s default output fields may be useless in some cases, causing redundancy. You can remove the redundant fields.

sudo vim /etc/filebeat/filebeat.yml

processors:
        #- add_host_metadata:
        #    when.not.contains.tags: forwarded
        #- add_cloud_metadata: ~
        #- add_docker_metadata: ~
        #- add_kubernetes_metadata: ~
  - drop_fields:
          fields: ["agent", "nginx", "log.offset", "fileset", "input"]

Add fields that need to be removed as needed.

Best Practices

Start the service

Start three components in sequence.

1
2
3

sudo service elasticsearch start
sudo service kibana start
sudo service filebeat start

Nginx site configuration kibana access

Using the secondary directory of the Nginx site as an access path, for example.

server {
    ...
    location /kibana/ {
            proxy_pass http://127.0.0.1:5601;
            rewrite ^/kibabna/(.*)$ /$1 break;
    }
}

Access to kibana is available through https://easeapi.com/kibana.

Configure password for kibana

1
2

apt install apache2-utils
htpasswd -c /home/htpasswd <username>

A password file based on HTTP Basic Authentication authentication will be generated. Add the following configuration to the Nginx site.

server {
    auth_basic 'easeapi center';
    auth_basic_user_file /home/htpasswd;
}

When the corresponding page is visited, it will prompt for password verification.

Adding indexes

After opening the kibana page, click “Add Data” and select “Nginx Logs” (there will be instructions for the Filebeat module to parse Nginx logs), and follow the instructions to finish.

Table of Contents