ELK日志收集部署
时间同步：1
2
ntpdate pool.ntp.org
echo ‘*/5 * * * * ntpdate pool.ntp.org’>> /var/spool/cron/root
关闭防火墙和selinux
1
2
3
/etc/init.d/iptablesstop
chkconfig iptables off
Sed –I ‘s/SELINUX=enforcing/SELINUX=disabled/g’/etc/selinux/config
安装java
1
2
3
4
5
6
7
8
9
10
11
12
13
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
tar zxf jdk-7u45-linux-x64.tar.gz
mv jdk1.8.0_92/ /usr/local/jdk
设置jdk环境变量
vi /etc/profile
-------------------------------------------------------
JAVA_HOME=/usr/local/jdk
PATH=$PATH:$JAVA_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export JAVA_HOME PATHCLASSPATH
-------------------------------------------------------
source /etc/profile
java -version
下载redis
1
2
3
4
5
6
7
8
9
10
wget
http://download.redis.io/releases/redis-3.2.3.tar.gz
tar zxf redis-3.2.3.tar.gz
cd redis-3.2.3
make
make PREFIX=/usr/local/redis install
mkdir /usr/local/redis/conf
cpredis.conf /usr/local/redis/conf/redis.conf.bak
cd /usr/local/redis/conf
cp redis.conf.bak redis.conf
添加环境变量
1
2
3
echo 'PATH=$PATH:/usr/local/redis/bin/' >>/etc/profilesource /etc/profile
启动redis：
/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf &
下载logstash elasticsearchkibana
1
2
3
4
https://www.elastic.co/downloads
elasticsearch-5.0.0.tar.gz
logstash-5.0.0.tar.gz
kibana-5.0.0-linux-x86_64.tar.gz
解压文件：
1
2
3
tar zxf logstash-5.0.0.tar.gz
tar zxf elasticsearch-5.0.0.tar.gz
tar zxf kibana-5.0.0-linux-x86_64.tar.gz
移动到统一管理目录：
1
2
3
mv elasticsearch-5.0.0 /usr/local/elasticsearch
mv logstash-5.0.0 /usr/local/logstash
mv kibana-5.0.0-linux-x86_64 /usr/local/kibana
备份配置文件：
1
2
3
cp /usr/local/logstash/config/logstash.yml /usr/local/logstash/config/logstash.yml.bak.$(date+%F)
cp /usr/local/elasticsearch/config/elasticsearch.yml/usr/local/elasticsearch/config/elasticsearch.yml.bak.$(date +%F)
cp /usr/local/kibana/config/kibana.yml /usr/local/kibana/config/kibana.yml.bak.$(date+%F)
配置Elasticsearch
创建用户
1
2
3
4
默认elasticsearch不支持root用户启动，所以需要先创建一个普通用户
groupadd elastic
useradd elastic –gelastic–M
chown -R elastic.elastic /usr/local/elasticsearch/
修改配置文件：
1
2
3
4
network.host: 192.168.0.248
http.port: 9200
su elastic
/usr/local/elasticsearch/bin/elasticsearch –d
验证启动：curl http://localhost:9200/添加开机启动：
1
echo ‘/usr/local/elasticsearch/bin/elasticsearch-d’ >>/etc/rc.local
注意错误：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
ERROR: bootstrap checks failed
问题：max file descriptors for elasticsearch process likelytoo low, increase to at least
解决：vi /etc/security/limits.conf
* - nofile 65536
或者
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
问题：max number of threads for user likely toolow, increase to at least
解决：vi/etc/security/limits.d/90-nproc.conf
* soft nproc 2048
需重启生效。
问题：max virtual memory areas vm.max_map_count likely toolow, increase to at least
解决：vi /etc/sysctl.conf
vm.max_map_count=655360
sysctl -p
配置kibana：
1
2
3
4
5
修改配置文件：
vi /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "192.168.0.248"
elasticsearch.url: http://192.168.0.248:9200
测试logstash：
1
2
3
4
cd logstash-5.0.0
bin/logstash -e 'input { stdin { } } output {stdout {} }'
hello world
2013-11-21T01:22:14.405+0000 0.0.0.0 helloworld
配置logstash服务端：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vi /usr/local/logstash/config/logstash.yml
input {
redis {
host => “192.168.0.248”
port => 6379
type =>”redis-input”
data_type =>”list”
key =>”logstash:redis”
}
}
output{
elasticsearch {
hosts=> ["192.168.0.248:9200"]
}
}
客户端：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
vi /usr/local/logstash/config/logstash.yml
input {
file{
type=>”nginx_access”
path=>[”/usr/local/nginx/logs/access.log”]
}
}
output {
redis{
host=> “192.168.0.248”
data_type=> “list”
key=> “logstash:redis”
}
}
启动客户端：
1
2
3
/usr/local/logstash/bin/logstash –f /usr/local/logstash/conf/logstash.conf
或yum安装：
/usr/share/logstash/bin/logstash -f/etc/logstash/conf.d/logstash.conf
客户端yum安装：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/logstash.repo <<EOF
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
yum clean all
yum install logstash –y
页:
[1]