Fluentd+ElasticSearch+Kibana 搭建日誌類文件分析器 安裝篇

zj2092

　　一 查看當前機器信息
　　1. 操作系統


　　

[coolbeans@coolbeans-fluentd ~]$ cd /proc

[coolbeans@coolbeans-fluentd proc]$ lsb_release -a

LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch

Distributor ID:    CentOS

Description:    CentOS release 6.4 (Final)

Release:    6.4

Codename:    Final
　　2. hostname
　　[coolbeans@coolbeans-fluentd ~]$ cat /proc/sys/kernel/hostname

coolbeans-fluentd

[coolbeans@coolbeans-fluentd ~]$ hostname

coolbeans-fluentd
　　3. 内存
　　[coolbeans@coolbeans-fluentd ~]$ free

             total       used       free     shared    buffers     cached

Mem:       1922464    1067420     855044          0     256628     230204

-/+ buffers/cache:     580588    1341876

Swap:      4128760          0    4128760

[coolbeans@coolbeans-fluentd ~]$ cat /proc/meminfo

MemTotal:        1922464 kB

MemFree:          854912 kB

Buffers:          256628 kB

Cached:           230204 kB

SwapCached:            0 kB

Active:           690624 kB

Inactive:         193840 kB

Active(anon):     397784 kB

Inactive(anon):        4 kB

Active(file):     292840 kB

Inactive(file):   193836 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:       4128760 kB

SwapFree:        4128760 kB

Dirty:                16 kB

Writeback:             0 kB

AnonPages:        397540 kB

Mapped:            26532 kB

Shmem:               164 kB

Slab:             150432 kB

SReclaimable:      91164 kB

SUnreclaim:        59268 kB

KernelStack:        1528 kB

PageTables:         4836 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:     5089992 kB

Committed_AS:     645828 kB

VmallocTotal:   34359738367 kB

VmallocUsed:      274520 kB

VmallocChunk:   34359447820 kB

HardwareCorrupted:     0 kB

AnonHugePages:    321536 kB

HugePages_Total:       0

HugePages_Free:        0

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

DirectMap4k:        8192 kB

DirectMap2M:     2088960 kB
　　4. CPU
　　[coolbeans@coolbeans-fluentd ~]$ cat /proc/cpuinfo | grep 'model name' |uniq

model name    : Intel(R) Xeon(R) CPU           X5660  @ 2.80GHz
　　5. Disk
　　[coolbeans@coolbeans-fluentd /]$ df -h

文件系统          容量  已用  可用 已用%% 挂载点

/dev/mapper/vg_coolbeansfluentd-lv_root

                       50G  3.1G   44G   7% /

tmpfs                 939M     0  939M   0% /dev/shm

/dev/sda1             485M   64M  397M  14% /boot

/dev/mapper/vg_coolbeansfluentd-lv_home

                      439G  1.6G  415G   1% /home


　　



二 查看Max # of File Descriptors
　　
　　ulimit -n
　　三 安裝git
　　安裝其所依賴之包：
　　yum install zlib-devel

yum install openssl-devel

yum install perl

yum install cpio

yum install expat-devel

yum install gettext-devel
　　yum install autoconf


　　//安裝GIT


　　wget http://www.codemonkey.org.uk/projects/git-snapshots/git/git-latest.tar.gz

tar xzvf git-latest.tar.gz

cd git-{date}

autoconf

./configure --with-curl=/usr/local

make

make install



四 安裝RUBY

下載ruby-2.0.0-p247.tar.gz

tar -zxvf ruby-2.0.0-p247.tar.gz

cd ruby-2.0.0-p247

./configure  -prefix=/opt/modules/ruby

make

make install

添加環境變量：export PATH=/opt/modules/ruby/bin:$PATH



測試RUBY GEM:

[coolbeans@coolbeans-fluentd ~]$ gem -v

2.0.3

備註：

在安裝RUBY前請確保OPEN-SSL已安裝

五 安裝elasticsearch

下載：elasticsearch-0.90.0.RC2.tar.gz

tar -zxvf elasticsearch-0.90.0.RC2.tar.gz

cd elasticsearch-0.90.0.RC2

啟動： ./bin/elasticsearch -f



六 安裝bundle

[coolbeans@coolbeans-fluentd bin]$ sudo ./gem install bundle

[sudo] password for coolbeans:

Fetching: bundler-1.3.5.gem (100%)

Successfully installed bundler-1.3.5

Fetching: bundle-0.0.1.gem (100%)

Successfully installed bundle-0.0.1

Parsing documentation for bundler-1.3.5

Installing ri documentation for bundler-1.3.5

Parsing documentation for bundle-0.0.1

Installing ri documentation for bundle-0.0.1

2 gems installed



七 安裝Kibana

[coolbeans@coolbeans-fluentd ~]$ cd Kibana-0.2.0

[coolbeans@coolbeans-fluentd Kibana-0.2.0]$ bundle install --path vendor/bundle

Fetching gem metadata from http://rubygems.org/.........

Fetching gem metadata from http://rubygems.org/..

Resolving dependencies...

Installing rake (10.1.0)

Installing daemons (1.1.9)

Installing diff-lcs (1.1.3)

Installing eventmachine (1.0.0)

Installing fastercsv (1.5.5)

Installing json (1.7.5)

Installing rack (1.4.1)

Installing rack-protection (1.2.0)

Installing tilt (1.3.3)

Installing sinatra (1.3.3)

Installing thin (1.5.0)

Installing tzinfo (0.3.35)

Using kibana (0.0.1) from source at .

Installing rspec-core (2.11.1)

Installing rspec-expectations (2.11.3)

Installing rspec-mocks (2.11.3)

Installing rspec (2.11.0)

Using bundler (1.3.5)

Your bundle is complete!

It was installed into ./vendor/bundle



啟動Kibana:


bundle exec ruby kibana.rb


默認新動態下，只可在本地訪問：
　　修改 KibanaConfig.rb  16行. 將 127.0.0.1 修改為 0.0.0.0 以綁定所有IP


　　# The adress ip Kibana should listen on

KibanaHost = '127.0.0.1'





八 安裝Fluented RPM:

執行該腳本即可

http://toolbelt.treasure-data.com/sh/install-redhat.sh



九 安裝ElasticSearch plugin for Fluentd: fluent-plugin-elasticsearch

/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch


[coolbeans@coolbeans-fluentd bin]$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch

[sudo] password for coolbeans:

Fetching: fluent-plugin-elasticsearch-0.1.2.gem (100%)

Successfully installed fluent-plugin-elasticsearch-0.1.2

1 gem installed

Installing ri documentation for fluent-plugin-elasticsearch-0.1.2...

Installing RDoc documentation for fluent-plugin-elasticsearch-0.1.2...



修改/etc/td-agent/td-agent.conf為：

<source>
type syslog
port 42185
tag syslog
</source>
<source>
type forward
</source>
<match syslog.**>
type elasticsearch
logstash_format true
flush_interval 10s # for testing
</match>


啟動td-agent
sudo /etc/init.d/td-agent start


十 fluent-plugin-notifier的安裝

     學前最新版本為：0.2.1

sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-notifier -v 0.2.1



十一  fluent-plugin-mail的安裝

sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mail



十二 fluent-plugin-tail-ex安裝

在in_tail plugin 的基礎上擴展，其主要新增功能為支持目錄文件收集

安裝命令：sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-tail-ex



十二  rsyslogd

系統已自帶