ELK日志分析平台
我的博客已迁移到xdoujiang.com请去那边和我交流ELKstack是Elasticsearch、Logstash、Kibana三个开源软件的组合。目前都在Elastic.co公司名下。
ELK是一套常用的开源日志监控和分析系统，包括一个分布式索引与搜索服务Elasticsearch，
一个管理日志和事件的工具logstash，和一个数据可视化服务Kibana
logstash_1.5.3 负责日志的收集，处理和储存
elasticsearch-1.7.2 负责日志检索和分析
kibana-4.1.2-linux-x64.tar.gz负责日志的可视化
jdk-1.7.0_03 java环境
redis-2.4.14 DB
一、基础环境
1、角色、ip、版本、内核
serverA 10.1.10.185 3.2.0-4-amd64 7.8 java elasticsearch redis kibana logstash(agent indexer)
clientB 10.1.10.117 3.2.0-4-amd64 7.8 java logstash(agent)
2、安装基础包
apt-get -y install curl wget lrzsz axel
二、安装redis server
1、安装包
apt-get -y install redis-server
2、创建redis存储目录
mkdir /opt/redis -p
3、权限
chown redis /opt/redis/ -R
4、配置
1)备份配置
cp /etc/redis/redis.conf /etc/redis/redis.conf.bak
2)修改配置
sed -i 's!^bind.*!bind 10.1.10.185!g' /etc/redis/redis.conf
sed -i 's!^dir.*!dir /opt/redis!g' /etc/redis/redis.conf
5、重启服务
/etc/init.d/redis-server restart
6、查看进程和端口
1)查看进程
ps -ef |grep redis
redis 23193 10 16:41 ? 00:00:00 /usr/bin/redis-server /etc/redis/redis.conf
2)查看端口
netstat -tupnl |grep redis
tcp 0 0 10.1.10.185:6379 0.0.0.0:* LISTEN 25188/redis-server
7、检查开机启动(默认设置开机启动了)
ll /etc/rc2.d/ |grep redis
lrwxrwxrwx 1 root root22 Sep 20 16:41 S02redis-server -> ../init.d/redis-server
三、安装java环境
1、安装包
apt-get -y install openjdk-7-jdk
2、查看版本
java -version
java version "1.7.0_03"
OpenJDK Runtime Environment (IcedTea7 2.1.7) (7u3-2.1.7-1)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
四、安装elasticsearch
1、下载elasticsearch
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb
2、安装elasticsearch
dpkg -i elasticsearch-1.7.2.deb
Selecting previously unselected package elasticsearch.
(Reading database ... 30240 files and directories currently installed.)
Unpacking elasticsearch (from elasticsearch-1.7.2.deb) ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Setting up elasticsearch (1.7.2) ...
3、配置
1)备份配置
cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
2)修改配置
echo "network.bind_host: 10.1.10.185" >> /etc/elasticsearch/elasticsearch.yml
4、启动elasticsearch服务
/etc/init.d/elasticsearch start
5、查看进程和端口
1)查看进程
ps -ef |grep java
106 22835 1 63 15:14 ? 00:00:03 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
2)查看端口
netstat -tupnl |grep java
tcp6 0 0 10.1.10.185:9200 :::* LISTEN 22835/java
tcp6 0 0 10.1.10.185:9300 :::* LISTEN 22835/java
udp6 0 0 :::54328 :::* 22835/java
6、测试
curl -X GET http://10.1.10.185:9200
{
"status" : 200,
"name" : "Ned Leeds",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.7.2",
"build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec",
"build_timestamp" : "2015-09-14T09:49:53Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
7、添加到开机启动
update-rc.d elasticsearch defaults
update-rc.d: using dependency based boot sequencing
五、安装logstash
1、下载logstash
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb
2、安装logstash
dpkg -i logstash_1.5.3-1_all.deb
(Reading database ... 30338 files and directories currently installed.)
Unpacking logstash (from logstash_1.5.3-1_all.deb) ...
Setting up logstash (1:1.5.3-1) ...
3、配置(默认没有这个配置文件)
1)配置logstash_agent
cat /etc/logstash/conf.d/logstash_agent.conf
input {
file {
type => "messages"
path => ["/var/log/messages"]
}
file {
type => "elasticsearch"
path => ['/var/log/elasticsearch/elasticsearch.log*']
}
}
output {
redis {
host => "10.1.10.185"
data_type => "list"
key => "logstash:redis"
}
}
2)配置logstash_indexer
cat /etc/logstash/conf.d/logstash_indexer.conf
input {
redis {
host => "10.1.10.185"
data_type => "list"
key => "logstash:redis"
type => "redis-input"
port => "6379"
}
}
output {
elasticsearch {
host => "10.1.10.185"
}
}
4、启动服务
/etc/init.d/logstash start
logstash started.
5、使用jps -mlv或ps -ef来查看下进程
ps -ef|grep logst
logstash22932 1 16 15:19 pts/0 00:00:01 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
6、设置开机启动
update-rc.d logstash defaults
update-rc.d: using dependency based boot sequencing
六、安装kibana(前端web)
1、下载
axel -n 10 https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
2、解压到指定目录
tar zxvf kibana-4.1.2-linux-x64.tar.gz -C /opt
3、创建日志目录
mkdir -p /opt/kibanalog
4、配置
1)备份配置
cp /opt/kibana-4.1.2-linux-x64/config/kibana.yml /opt/kibana-4.1.2-linux-x64/config/kibana.yml.bak
2)修改配置
sed -i 's!^elasticsearch_url: .*!elasticsearch_url: "http://10.1.10.185:9200"!g' /opt/kibana-4.1.2-linux-x64/config/kibana.yml
sed -i 's!^host: .*!host: "10.1.10.185"!g' /opt/kibana-4.1.2-linux-x64/config/kibana.yml
5、启动服务
cd /opt/kibanalog && nohup /opt/kibana-4.1.2-linux-x64/bin/kibana &
6、查看进程和端口
1)查看进程
ps aux |grep kibana
root 229825.4 20.1 612576 47716 pts/0 Sl 15:22 0:01 /opt/kibana-4.1.2-linux-x64/bin/../node/bin/node /opt/kibana-4.1.2-linux-x64/bin/../src/bin/kibana.js
2)查看端口
netstat -tupnl|grep 5601
tcp 0 0 10.1.10.185:5601 0.0.0.0:* LISTEN 22982/node
7、在windows上访问http://10.1.10.185:5601http://s3.运维网.com/wyfs02/M01/73/98/wKiom1YB9GrwvtzrAANcUZu8rDw596.jpg
http://s3.运维网.com/wyfs02/M01/73/95/wKioL1YB9GyzXKWkAAbO4sll4MI251.jpg
8、设置开机启动
echo "cd /opt/kibanalog && nohup /opt/kibana-4.1.2-linux-x64/bin/kibana &" >> /etc/rc.local
七、查看全部服务日志
1、查看redis日志
cat /var/log/redis/redis-server.log
22 Sep 09:53:47 * Server started, Redis version 2.4.14
22 Sep 09:53:47 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
22 Sep 09:53:47 * The server is now ready to accept connections on port 6379
22 Sep 10:08:42 # Received SIGTERM, scheduling shutdown...
22 Sep 10:08:42 # User requested shutdown...
22 Sep 10:08:42 * Saving the final RDB snapshot before exiting.
22 Sep 10:08:42 * DB saved on disk
22 Sep 10:08:42 * Removing the pid file.
22 Sep 10:08:42 # Redis is now ready to exit, bye bye...
22 Sep 10:08:43 * Server started, Redis version 2.4.14
22 Sep 10:08:43 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
22 Sep 10:08:43 * The server is now ready to accept connections on port 6379
22 Sep 10:49:56 * 1 changes in 900 seconds. Saving...
22 Sep 10:49:56 * Background saving started by pid 23186
22 Sep 10:49:56 * DB saved on disk
22 Sep 10:49:57 * Background saving terminated with success
22 Sep 10:50:58 * 10000 changes in 60 seconds. Saving...
22 Sep 10:50:58 * Background saving started by pid 23205
22 Sep 10:50:58 * DB saved on disk
22 Sep 10:50:58 * Background saving terminated with success
22 Sep 10:51:59 * 10000 changes in 60 seconds. Saving...
22 Sep 10:51:59 * Background saving started by pid 23214
22 Sep 10:51:59 * DB saved on disk
22 Sep 10:51:59 * Background saving terminated with success
2、查看elasticsearch日志
cat /var/log/elasticsearch/elasticsearch.log
version, pid, build
initializing ...
loaded [], sites []
using data paths, mounts [[/ (rootfs)]], net usable_space , net total_space , types
initialized
starting ...
bound_address {inet}, publish_address {inet}
elasticsearch/mB1_wQprTAWGam7X1LzCxQ
new_master ], reason: zen-disco-join (elected_as_master)
bound_address {inet}, publish_address {inet}
started
recovered indices into cluster_state
added {]{client=true, data=false},}, reason: zen-disco-receive(join from node[]{client=true, data=false}])
creating index, cause , templates , shards /, mappings
update_mapping (dynamic)
observer: timeout notification from cluster service. timeout setting , time since start
[.kibana] creating index, cause , templates [], shards /, mappings []
[.kibana] update_mapping (dynamic)
3、查看logstash日志
cat /var/log/logstash/logstash.err
!!! Please upgrade your java version, the current version '1.7.0_03-b21' may cause problems. We recommend a minimum version of 1.7.0_51
' use `require concurrent` instead of `require concurrent_ruby'
WARN -- Concurrent: Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).
Sep 22, 2015 10:49:49 AM org.elasticsearch.node.internal.InternalNode
INFO: version, pid, build
Sep 22, 2015 10:49:49 AM org.elasticsearch.node.internal.InternalNode
INFO: initializing ...
Sep 22, 2015 10:49:49 AM org.elasticsearch.plugins.PluginsService
INFO: loaded [], sites []
Sep 22, 2015 10:49:51 AM org.elasticsearch.bootstrap.Natives
WARNING: JNA not found. native methods will be disabled.
Sep 22, 2015 10:49:52 AM org.elasticsearch.node.internal.InternalNode
INFO: initialized
Sep 22, 2015 10:49:52 AM org.elasticsearch.node.internal.InternalNode start
INFO: starting ...
Sep 22, 2015 10:49:52 AM org.elasticsearch.transport.TransportService doStart
INFO: bound_address {inet}, publish_address {inet}
Sep 22, 2015 10:49:53 AM org.elasticsearch.discovery.DiscoveryService doStart
INFO: elasticsearch/Bt8LxnD9R4amhOypJbgxww
Sep 22, 2015 10:49:56 AM org.elasticsearch.cluster.service.InternalClusterService$UpdateTask run
INFO: detected_master ], added {],}, reason: zen-disco-receive(from master []])
Sep 22, 2015 10:49:56 AM org.elasticsearch.node.internal.InternalNode start
INFO: started
Sep 22, 2015 10:55:23 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
WARNING: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
Sep 22, 2015 10:56:08 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
INFO: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
Sep 22, 2015 10:57:39 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
INFO: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
Sep 22, 2015 11:00:42 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
INFO: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
Sep 22, 2015 11:01:33 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
WARNING: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
Sep 22, 2015 11:02:17 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
WARNING: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
Sep 22, 2015 11:04:40 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
WARNING: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
Sep 22, 2015 11:05:30 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
WARNING: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
Sep 22, 2015 11:06:16 AM org.elasticsearch.monitor.jvm.JvmMonitorService$JvmMonitor monitorLongGc
INFO: duration , collections /, total /, memory ->/, all_pools { ->/}{ ->/}{ ->/}
4、查看kibana日志
cat /opt/kibanalog/nohup.out
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"msg":"No existing kibana index found","time":"2015-09-22T02:53:28.503Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"msg":"Listening on 10.1.10.185:5601","time":"2015-09-22T02:53:28.538Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:29 GMT","if-none-match":"W/\"6f9-3043805189\""},"remoteAddress":"10.1.10.131","remotePort":57468},"res":{"statusCode":304,"responseTime":6,"contentLength":0},"msg":"GET / 304 - 6ms","time":"2015-09-22T02:53:49.894Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/main.css?_b=7562","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/css,*/*;q=0.1","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"335dc-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57468},"res":{"statusCode":304,"responseTime":18,"contentLength":0},"msg":"GET /styles/main.css?_b=7562 304 - 18ms","time":"2015-09-22T02:53:49.964Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/images/initial_load.gif","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"image/png,image/*;q=0.8,*/*;q=0.5","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:29 GMT","if-none-match":"W/\"2e9e-3043805189\""},"remoteAddress":"10.1.10.131","remotePort":57468},"res":{"statusCode":304,"responseTime":2,"contentLength":0},"msg":"GET /images/initial_load.gif 304 - 2ms","time":"2015-09-22T02:53:49.968Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/bower_components/requirejs/require.js?_b=7562","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"*/*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:29 GMT","if-none-match":"W/\"14703-3043805189\""},"remoteAddress":"10.1.10.131","remotePort":57469},"res":{"statusCode":304,"responseTime":1,"contentLength":0},"msg":"GET /bower_components/requirejs/require.js?_b=7562 304 - 1ms","time":"2015-09-22T02:53:49.969Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/require.config.js?_b=7562","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"*/*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:29 GMT","if-none-match":"W/\"a66-3043805189\""},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":304,"responseTime":2,"contentLength":0},"msg":"GET /require.config.js?_b=7562 304 - 2ms","time":"2015-09-22T02:53:49.970Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/theme/elk.ico","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"47e-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57471},"res":{"statusCode":304,"responseTime":3,"contentLength":0},"msg":"GET /styles/theme/elk.ico 304 - 3ms","time":"2015-09-22T02:53:49.970Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/index.js?_b=7562","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"*/*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:29 GMT","if-none-match":"W/\"5489a7-3043805189\""},"remoteAddress":"10.1.10.131","remotePort":57472},"res":{"statusCode":304,"responseTime":1,"contentLength":0},"msg":"GET /index.js?_b=7562 304 - 1ms","time":"2015-09-22T02:53:50.037Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/theme/elk.ico","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"47e-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57468},"res":{"statusCode":304,"responseTime":0,"contentLength":0},"msg":"GET /styles/theme/elk.ico 304 - 0ms","time":"2015-09-22T02:53:50.203Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/config?_b=7562","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive","if-none-match":"W/\"151-5c053bf3\""},"remoteAddress":"10.1.10.131","remotePort":57469},"res":{"statusCode":304,"responseTime":3,"contentLength":0},"msg":"GET /config?_b=7562 304 - 3ms","time":"2015-09-22T02:53:50.492Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/elasticsearch/?_=1442890430562","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":200,"responseTime":12,"contentLength":333},"msg":"GET /?_=1442890430562 200 - 12ms","time":"2015-09-22T02:53:50.883Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/theme/elk.ico","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"47e-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57471},"res":{"statusCode":304,"responseTime":0,"contentLength":0},"msg":"GET /styles/theme/elk.ico 304 - 0ms","time":"2015-09-22T02:53:50.923Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/theme/elk.ico","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"47e-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57472},"res":{"statusCode":304,"responseTime":0,"contentLength":0},"msg":"GET /styles/theme/elk.ico 304 - 0ms","time":"2015-09-22T02:53:50.926Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/theme/elk.ico","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"47e-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57468},"res":{"statusCode":304,"responseTime":1,"contentLength":0},"msg":"GET /styles/theme/elk.ico 304 - 1ms","time":"2015-09-22T02:53:50.929Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/elasticsearch/_nodes?_=1442890430791","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":200,"responseTime":63,"contentLength":5943},"msg":"GET /_nodes?_=1442890430791 200 - 63ms","time":"2015-09-22T02:53:51.153Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"HEAD","url":"/elasticsearch/.kibana","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":404,"responseTime":6,"contentLength":0},"msg":"HEAD /.kibana 404 - 6ms","time":"2015-09-22T02:53:51.171Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"POST","url":"/elasticsearch/.kibana","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","content-type":"application/json;charset=utf-8","referer":"http://10.1.10.185:5601/","content-length":"35","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":200,"responseTime":546,"contentLength":21},"msg":"POST /.kibana 200 - 546ms","time":"2015-09-22T02:53:51.726Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/elasticsearch/_cluster/health/.kibana?wait_for_status=yellow&_=1442890431632","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":200,"responseTime":467,"contentLength":313},"msg":"GET /_cluster/health/.kibana?wait_for_status=yellow&_=1442890431632 200 - 467ms","time":"2015-09-22T02:53:52.398Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"POST","url":"/elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1442890430199","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","content-type":"application/json;charset=utf-8","referer":"http://10.1.10.185:5601/","content-length":"62","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":200,"responseTime":60,"contentLength":76},"msg":"POST /_mget?timeout=0&ignore_unavailable=true&preference=1442890430199 200 - 60ms","time":"2015-09-22T02:53:52.665Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"POST","url":"/elasticsearch/.kibana/config/4.1.2","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","content-type":"application/json;charset=utf-8","referer":"http://10.1.10.185:5601/","content-length":"17","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":201,"responseTime":33,"contentLength":79},"msg":"POST /.kibana/config/4.1.2 201 - 33ms","time":"2015-09-22T02:53:52.908Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"POST","url":"/elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1442890430199","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","content-type":"application/json;charset=utf-8","referer":"http://10.1.10.185:5601/","content-length":"62","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":200,"responseTime":6,"contentLength":116},"msg":"POST /_mget?timeout=0&ignore_unavailable=true&preference=1442890430199 200 - 6ms","time":"2015-09-22T02:53:53.126Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"POST","url":"/elasticsearch/.kibana/index-pattern/_search?fields=","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","content-type":"application/json;charset=utf-8","referer":"http://10.1.10.185:5601/","content-length":"44","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":200,"responseTime":152,"contentLength":124},"msg":"POST /.kibana/index-pattern/_search?fields= 200 - 152ms","time":"2015-09-22T02:53:53.529Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/theme/elk.ico","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"47e-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57469},"res":{"statusCode":304,"responseTime":1,"contentLength":0},"msg":"GET /styles/theme/elk.ico 304 - 1ms","time":"2015-09-22T02:53:54.035Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/theme/elk.ico","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"47e-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57471},"res":{"statusCode":304,"responseTime":0,"contentLength":0},"msg":"GET /styles/theme/elk.ico 304 - 0ms","time":"2015-09-22T02:53:54.036Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/styles/theme/elk.ico","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:30 GMT","if-none-match":"W/\"47e-873763449\""},"remoteAddress":"10.1.10.131","remotePort":57472},"res":{"statusCode":304,"responseTime":9,"contentLength":0},"msg":"GET /styles/theme/elk.ico 304 - 9ms","time":"2015-09-22T02:53:54.051Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/elasticsearch/logstash-*/_mapping/field/*?ignore_unavailable=false&allow_no_indices=false&include_defaults=true&_=1442890433545","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/json, text/plain, */*","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/","connection":"keep-alive"},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":200,"responseTime":235,"contentLength":11412},"msg":"GET /logstash-*/_mapping/field/*?ignore_unavailable=false&allow_no_indices=false&include_defaults=true&_=1442890433545 200 - 235ms","time":"2015-09-22T02:53:54.096Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/images/no_border.png","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"image/png,image/*;q=0.8,*/*;q=0.5","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate","referer":"http://10.1.10.185:5601/styles/main.css?_b=7562","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:29 GMT","if-none-match":"W/\"10ab-3043805189\""},"remoteAddress":"10.1.10.131","remotePort":57470},"res":{"statusCode":304,"responseTime":2,"contentLength":0},"msg":"GET /images/no_border.png 304 - 2ms","time":"2015-09-22T02:53:54.377Z","v":0}
{"name":"Kibana","hostname":"debian","pid":23238,"level":30,"req":{"method":"GET","url":"/bower_components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0","headers":{"host":"10.1.10.185:5601","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0","accept":"application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"identity","referer":"http://10.1.10.185:5601/styles/main.css?_b=7562","connection":"keep-alive","if-modified-since":"Tue, 08 Sep 2015 20:12:29 GMT","if-none-match":"W/\"ffac-3043805189\""},"remoteAddress":"10.1.10.131","remotePort":57468},"res":{"statusCode":304,"responseTime":1,"contentLength":0},"msg":"GET /bower_components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0 304 - 1ms","time":"2015-09-22T02:53:54.378Z","v":0}
八、clientB安装配置logstash(agent)
1、安装java环境
apt-get -y install openjdk-7-jdk
2、下载logstash
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb
3、安装logstash
dpkg -i logstash_1.5.3-1_all.deb
(Reading database ... 30338 files and directories currently installed.)
Unpacking logstash (from logstash_1.5.3-1_all.deb) ...
Setting up logstash (1:1.5.3-1) ...
4、配置(默认没有这个配置文件)
1)配置logstash_agent
cat /etc/logstash/conf.d/logstash_agent.conf
input {
file {
type => "message"
path => ["/var/log/message'"]
}
}
output {
redis {
host => "10.1.10.185"
data_type => "list"
key => "logstash:redis"
}
}
5、启动服务
/etc/init.d/logstash start
logstash started.
6、使用jps -mlv或ps -ef来查看下进程
ps -ef|grep logst
logstash22932 1 16 15:19 pts/0 00:00:01 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
7、设置开机启动
update-rc.d logstash defaults
update-rc.d: using dependency based boot sequencing
九、参考文章:
https://www.elastic.co/products
http://wsgzao.github.io/post/elk/
页:
[1]