elk安装过程

目的

使用ELK集中管理日志，如集群应用中所有节点的日志放在一起查看，并按日志级别分别过滤，实时监控错误日志，以及对业务一些数据进行分析，如实时车辆位置展示等等。

版本选择

作为尝鲜者，选择官网最新版本来尝试

• ElasticSearch5.4.0
• Filebeat5.4.0
• Logstash5.4.0
• Kibana5.4.0
• Redis3.2.9
• x-pack
• nginx1.12.0
• Centos7.3 x64

各组件的简略架构图

ElasticSearch安装过程

• 按照官网的操作步骤安装十分简单，第一步，下载并解压，第二步，运行bin/elasticsearch 就完事了
• 官网地址：(https://www.elastic.co/downloads/elasticsearch)
• 实际在运行的时候可以使用守护方式运行，bin/elasticsearch -d
• 安装过程中遇到的几个错误和解决办法

第一个错误：不能使用root用户启动（can not run elasticsearch as root）

[root@appsvr02 bin]# ./elasticsearch
[2017-05-24T09:19:11,866][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.4.0.jar:5.4.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:106) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0]
        ... 6 more

因此，添加elk帐户： useradd elk， 并把原来目录所有都改为 elk, chown -R 命令。

---

第二个错误：内核版本不满足要求(seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled)

[elk@lnbAppsvr02 bin]$ ./elasticsearch
[2017-05-24T09:20:41,418][WARN ][o.e.b.JNANatives         ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
        at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:350) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:215) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:99) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:204) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]

通过官网论坛了解到，这个并不影响，可以不关心，不开启system call功能即可，当然，我是为了搭建生产环境，我重新选择了CentOS7.3,内核就是3.10+了，启动时也不会再有这个错误提示了。

第三个错误：系统参数配置不满足(max file descriptors，max number of threads，vm.max_map_count)

> max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
max number of threads [1024] for user [elk] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

解决办法，修改系统内核参数，
1.在/etc/sysctl.conf中增加 vm.max_map_count=655360；
2.在/etc/security/limit.conf中修改如下参数

# End of file
root soft nofile 1048576
root hard nofile 1048576
* soft nofile 1048576
* hard nofile 1048576

3.在/etc/security/limit.d/20-nproc.conf中修改* soft nproc 1024为10240，如下所示

*          soft    nproc     10240
root       soft    nproc     unlimited

• 配置文件修改，config/elasticsearch.yml

  http.port: 9200              ###按需进行修改
  network.host: 127.0.0.1      ###绑定指定访问IP,如果只允许内网访问则绑定内网IP

Kibana安装过程

• 官网下载安装包(https://www.elastic.co/downloads/kibana)

  curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz
  tar zxvf kibana-5.4.0-linux-x86_64.tar.gz

• 配置config/kibana.yml中elasticsearch.url，elasticsearch.username，elasticsearch.password参数

• 运行 bin/kibana &

X-Pack安装过程

• 不需要下载，直接作为插件安装在elasticsearch和kibana中
• 安装x-pack后登录kibana需要密码，默认密码elastic / changeme

  bin/elasticsearch-plugin install x-pack
  bin/kibana-plugin install x-pack

Redis3.2.9安装过程

• 官网下载最新版本（https://redis.io/download）

• 通过命令下载，解压并编译

  curl -L -O http://download.redis.io/releases/redis-3.2.9.tar.gz
  tar zxvf redis-3.2.9.tar.gz
  cd redis-3.2.9
  make
  ... then copy binary files with prefix redis- to /usr/local/redis/bin

• 修改配置参数，如port,bind,logfile,logpath,dumpfilename,requirepass,rename，如下：

  bind 10.xx.xx.xxx 127.0.0.1          ###仅允许内网和本机访问
  protected-mode yes                   ###保护模式开启
  port 123456                          ###端口默认为6379，按需修改
  daemonize yes                        ###守护模式开启
  pidfile /usr/local/redis/redis.pid   ###指定pid文件路径和文件名
  logfile "/usr/local/redis/redis.log" ###指定日志文件路径和文件名
  dbfilename redis.rdb                 ###指定数据文件RDB文件名
  dir /usr/local/redis/                ###指定数据文件RDB文件的存放路径
  requirepass 『YOURPASSWORD』          ###设置访问密码
  rename-command CONFIG "OTHERALIAS-0" ###把不安全的命令进行重命名保护
  rename-command FLUSHDB "OTHERALIAS-1"
  rename-command FLUSHALL "OTHERALIAS-2"

• REDIS启动

  /usr/local/bin/redis-server /usr/local/bin/redis.conf &

Logstash安装过程

• 官网下载安装包 (https://www.elastic.co/downloads/logstash)

  curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-5.4.0.tar.gz
  tar zxvf logstash-5.4.0.tar.gz

• 增加indexer配置

  input {
      redis {
          host => "127.0.0.1"
          type => "redis-input"
          data_type => "list"
          key => "logindexer_list"
          port => 36469
          password => "YOUR REDIS PASSWORD"
      }
  }
  # filter configration here
  output {
      elasticsearch {
           hosts => ["127.0.0.1:9200"]
           user => elastic
           password => xxyyzz...
      }
  }

Filebeat安装过程

• filebeat安装在各台应用服务器上，收集相关的日志内容输出到redis队列中
• 官网下载安装(https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.0-linux-x86_64.tar.gz)

• 分发到应用服务器上安装

  tar zxvf filebeat-5.4.0-linux-x86_64.tar.gz

• 复制filebeat.yml为filebeat.lnbapp.yml

• 配置文件位置和输出到redis相关配置

  -input_type: log
    #Paths that should be crawled and fetched. Glob based paths.
    paths:
      #- /var/log/*.log
      #- c:\programdata\elasticsearch\logs\*
      - /appserver/tomcat/logs/catalina.out
  	- /..../.../.../logfile
  	### 多个路径按模板往下添加即可
      #------------------- Redis output --------------------------
  output.redis:
    enabled: true
    hosts: ["10.xx.xx.xx:6379"]
    key: logindexer_list
    password: YOUR REDIS PASSWORD

• 启动filebeat

  bin/filebeat -e -c filebeat.lnbapp.yml &

Nginx安装过程

• 反向代理，作为整个日志集中化的对外接口

• 下载编译安装

  curl -L -O http://nginx.org/download/nginx-1.12.0.tar.gz
  tar zxvf nginx-1.12.0.tar.gz
  yum -y install pcre-devel openssl openssl-devel
  configure prefix=/usr/local/nginx --with-http_ssl_module
  make && make install

• 配置反向代理

  server {
          listen       80;
          server_name  elastic.jinsk.vip;
          #charset koi8-r;
          access_log  logs/elastic.access.log  main;
          location / {
              #root   html;
              #index  index.html index.htm;
                  proxy_pass http://127.0.0.1:5601/;
                  proxy_set_header        Host $host;
                  proxy_set_header        X-Real-IP $remote_addr;
                  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
          }
  }

到此，通过http://elastic.jinsk.vip打开登录界面

输入用户名，密码登录进入主界面