ELK日志收集部署

432edw

时间同步：

1 2	ntpdate pool.ntp.org echo ‘*/5 * * * * ntpdate pool.ntp.org’>> /var/spool/cron/root

关闭防火墙和selinux

1 2 3

/etc/init.d/iptables  stop chkconfig iptables off Sed –I ‘s/SELINUX=enforcing/SELINUX=disabled/g’  /etc/selinux/config

安装java

1 2 3 4 5 6 7 8 9 10 11 12 13

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html tar zxf jdk-7u45-linux-x64.tar.gz mv jdk1.8.0_92/ /usr/local/jdk 设置jdk环境变量 vi /etc/profile ------------------------------------------------------- JAVA_HOME=/usr/local/jdk PATH=$PATH:$JAVA_HOME/bin CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export JAVA_HOME PATHCLASSPATH ------------------------------------------------------- source /etc/profile java -version

下载redis

1 2 3 4 5 6 7 8 9 10

wget http://download.redis.io/releases/redis-3.2.3.tar.gz tar zxf redis-3.2.3.tar.gz cd redis-3.2.3 make make PREFIX=/usr/local/redis install mkdir /usr/local/redis/conf cp  redis.conf /usr/local/redis/conf/redis.conf.bak cd /usr/local/redis/conf cp redis.conf.bak redis.conf

添加环境变量

1 2 3

echo 'PATH=$PATH:/usr/local/redis/bin/' >>/etc/profilesource /etc/profile 启动redis： /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf &

下载logstash elasticsearch  kibana

1 2 3 4

https://www.elastic.co/downloads elasticsearch-5.0.0.tar.gz         logstash-5.0.0.tar.gz kibana-5.0.0-linux-x86_64.tar.gz

解压文件：

1 2 3

tar zxf logstash-5.0.0.tar.gz tar zxf elasticsearch-5.0.0.tar.gz tar zxf kibana-5.0.0-linux-x86_64.tar.gz

移动到统一管理目录：

1 2 3

mv elasticsearch-5.0.0 /usr/local/elasticsearch mv logstash-5.0.0 /usr/local/logstash mv kibana-5.0.0-linux-x86_64 /usr/local/kibana

备份配置文件：

1 2 3

cp /usr/local/logstash/config/logstash.yml /usr/local/logstash/config/logstash.yml.bak.$(date+%F)       cp /usr/local/elasticsearch/config/elasticsearch.yml/usr/local/elasticsearch/config/elasticsearch.yml.bak.$(date +%F) cp /usr/local/kibana/config/kibana.yml /usr/local/kibana/config/kibana.yml.bak.$(date+%F)

配置Elasticsearch

创建用户

1 2 3 4

默认elasticsearch不支持root用户启动，所以需要先创建一个普通用户 groupadd elastic useradd elastic –g  elastic  –M chown -R elastic.elastic   /usr/local/elasticsearch/

修改配置文件：

1 2 3 4

network.host: 192.168.0.248 http.port: 9200 su elastic /usr/local/elasticsearch/bin/elasticsearch –d

验证启动：

curl http://localhost:9200/

添加开机启动：

1	echo ‘/usr/local/elasticsearch/bin/elasticsearch-d’ >>/etc/rc.local

注意错误：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

ERROR: bootstrap checks failed 问题：max file descriptors [65535] for elasticsearch process likelytoo low, increase to at least [65536] 解决：vi /etc/security/limits.conf *             -       nofile          65536 或者 * soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096 问题：max number of threads [1024] for user [elasticsearch] likely toolow, increase to at least [2048] 解决：vi/etc/security/limits.d/90-nproc.conf * soft nproc 2048 需重启生效。 问题：max virtual memory areas vm.max_map_count [65530] likely toolow, increase to at least [262144] 解决：vi /etc/sysctl.conf vm.max_map_count=655360 sysctl -p

配置kibana：

1 2 3 4 5

修改配置文件： vi /usr/local/kibana/config/kibana.yml server.port: 5601 server.host: "192.168.0.248" elasticsearch.url: http://192.168.0.248:9200

测试logstash：

1 2 3 4

cd logstash-5.0.0 bin/logstash -e 'input { stdin { } } output {stdout {} }' hello world 2013-11-21T01:22:14.405+0000 0.0.0.0 helloworld

配置logstash服务端：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

vi /usr/local/logstash/config/logstash.yml input {          redis {                    host => “192.168.0.248”                    port => 6379                    type =>”redis-input”                    data_type =>”list”                    key =>”logstash:redis” } } output{ elasticsearch {         hosts=> ["192.168.0.248:9200"]            }         }

客户端：

1 2 3 4 5 6 7 8 9 10 11 12 13 14

vi /usr/local/logstash/config/logstash.yml input {          file{                    type=>”nginx_access”                    path=>[”/usr/local/nginx/logs/access.log”] } } output {          redis{                    host  => “192.168.0.248”                    data_type=> “list”                    key=> “logstash:redis” } }

启动客户端：

1 2 3

/usr/local/logstash/bin/logstash –f /usr/local/logstash/conf/logstash.conf 或yum安装： /usr/share/logstash/bin/logstash -f/etc/logstash/conf.d/logstash.conf

客户端yum安装：

1 2 3 4 5 6 7 8 9 10 11 12 13 14

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch cat > /etc/yum.repos.d/logstash.repo <<EOF [logstash-5.x] name=Elastic repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF    yum clean all yum install logstash –y