Elastic Stack5.2.2升级到6.0.0注意事项

sxyzy

最近把Elastic Stack从5.2.2版本升级到6.0.0版本，性能确实有所提高，文档记录了升级过程中需要注意的一些问题。

架构图

一、Filebeat
　　6.0版本filebeat prospectors中的document_type被禁用,原来的topic: '%{[type]}'获取文档类型的方式不可用，但是引入了fields，后面映射模板也会用到。
1、document_type: yewuname  原来的Index名字为filebeat-yewuname-2018.01.19这样的按照天分割的，升级后document_type不支持了，需要修改为字段加type的形式如下fields: document_type: yewuname
2、输出kafka也需要修改topic: '%{[fields.document_type]}'
3、logstash生成index也需要由原来的index => "filebeat-%{type}-%{+YYYY.MM.dd}"改为index => "filebeat-%{[fields][document_type]}-%{+YYYY.MM.dd}"，最终index命名为filebeat-yewuname-*
4、最后和Elasic支持人员沟通后得知，客户端filebeat不升级也可以，不影响使用

二、Logstash
　　1、主要是自定义映射模板需要修改，模板这块儿改动比较大，可以先使用默认的获取格式后再修改，然后得出自己的模板运用，使用了geoip定位，相关类型需要修改，最终模板如下：

cat /etc/logstash/templates/nginx_template
{
"template" : "filebeat-*",
"settings" : {
"index.refresh_interval" : "5s",
"index.number_of_shards": "8",
"index.number_of_replicas": "1"
},
"mappings": {
"_default_": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"agent": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"beat": {
"type": "object"
},
"clientRealIp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"geoip": {
"properties": {
"city_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"continent_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code2": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dma_code": {
"type": "long"
},
"ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"latitude": {
"type": "float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "float"
},
"postal_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_code": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"http_host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"method": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"referrer": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"request_uri": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"responsetime": {
"type": "float"
},
"size": {
"type": "long"
},
"status": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"upstreamhost": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"url": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
　　2、输出到elasticsearch有些参数不支持了，需要注释flush_size,否则会报错The setting flush_size in plugin elasticsearch is obsolete and is no longer available. This setting is no longer available as we now try to restrict bulk requests to sane sizes. See the 'Batch Sizes' section of the docs.

output {   
elasticsearch {
hosts => ["192.168.88.240:9200","192.168.88.241:9200","192.168.88.242:9200","192.168.88.243:9200","192.168.88.244:9200"]
index => "filebeat-%{[fields][document_type]}-%{+YYYY.MM.dd}"
manage_template => true
template_overwrite => true
template_name => "nginx_template"
template => "/etc/logstash/templates/nginx_template"
#flush_size => 20000
#idle_flush_time => 5
}
}
三、Elasticsearch
　　主要变更是该版本的一个Index只支持一个type了
官方提供有滚动升级，按照这个步骤尝试了下没有成功，后删数据硬升级，如果大家想尝试，步骤如下
滚动升级
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/rolling-upgrades.html
　　1、Disable shard allocation
例子：

curl -XPUT '192.168.88:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.allocation.enable": "none"
}
}
'
　　例子：

curl -XPUT '192.168.88:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
> {
>   "transient": {
>     "cluster.routing.allocation.enable": "none"
>   }
> }
> '
{
"acknowledged" : true,
"persistent" : { },
"transient" : {
"cluster" : {
"routing" : {
"allocation" : {
"enable" : "none"
}
}
}
}
}
　　2、Stop non-essential indexing and perform a synced flush (Optional)
curl -XPOST '192.168.88:9200/_flush/synced?pretty'
可选的，忽略
　　3、Stop and upgrade a single node
/etc/init.d/elasticsearch stop
rpm -e elasticsearch-5.2.2-1.noarch
rpm -ivh elasticsearch-6.0.0.rpm
chkconfig --add elasticsearch
chkconfig elasticsearch on
cp elasticsearch.yml.rpmsave  elasticsearch.yml
注意权限，卸载rpm包再安装rpm包后elasticsearch的uid与gid会变化
chown elasticsearch:elasticsearch /data/eslog/ -R
chown elasticsearch:elasticsearch /data/esngx1/ -R
chown elasticsearch:elasticsearch /data/esngx2/ -R
　　4、Upgrade any plugins
没有第三方插件，忽略
　　5、Start the upgraded node
curl -XGET '192.168.88:9200/_cat/nodes?pretty'
　　6、Reenable shard allocation

curl -XPUT '192.168.88:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
"transient": {
"cluster.routing.allocation.enable": "all"
}
}
'
　　7、Wait for the node to recover
curl -XGET '192.168.88:9200/_cat/health?pretty'
curl -XGET '192.168.88:9200/_cat/recovery?pretty'
　　8、Repeat

四、Kibana
　　1、index名字是字段type加通配
2、导出老visualize模板后所有的visualize里面.raw改成.keyword，然后导入时dashboard需要与index一一对应

五、Grafana
　　同样修改数据源的index名字与查询语法.raw关键字为.keyword即可

六、Elastalert
　　由于https://github.com/Yelp/elastalert/releases  安装部署是最新版本是v0.1.25还不支持elasticsearch6.0，后参考https://github.com/Yelp/elastalert/pull/1426 修
　　改/root/elastalert-0.1.25/elastalert目录下elastalert.py、create_index.py、ruletypes.py、test_rule.py四个文件，最后安装（备注以后安装后可以直接复制该目录到其他服务器上
　　安装，不需要再修改源码）
现在v0.1.26版本更新就支持elasticsearch6.0了，直接更新新版本就行。

七、其他的kafka、nginx等不需要调整。