|
|
ELK简单安装与配置
Elasticsearch是一个分布式可扩展的实时搜索和分析引擎，它能帮助你搜索、分析和浏览数据；它是一个建立在全文搜索引擎 Apache Lucene(TM) 基础上的搜索引擎，可以说Lucene是当今最先进，最高效的全功能开源搜索引擎框架。
一、环境介绍
1、架构介绍
ELK是C/S架构的，所以这里列出server和client
server : centos 6.6 x86_64 IP: 10.0.90.24
client : centos 6.6 x86_64 IP: 10.0.90.25
安装方式：这里采用源码包安装，也可以使用rpm包安装！
2、软件介绍
服务端软件：
Elasticsearch：负责日志检索和分析，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等
Logstash：对日志进行收集、过滤，并将其存储供以后使用（如，搜索日志）
Kibana：为日志分析提供友好的Web界面，可以帮助汇总、分析和搜索重要数据日志
客户端软件：
在需要收集日志的所有服务上部署logstash，作为logstash agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到logstash indexer，logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索，然后通过Kibana来结合自定义搜索进行页面展示。
原理图：摘自网络，连接忘记了，抱歉！
二、开始在server端安装配置ELK
1、安装jdk（下载地址http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html）
由于Logstash的运行依赖于Java环境， 而Logstash1.5以上版本依赖的java版本不低于java 1.7，因此推荐使用最新版本的Java。
我们只需要Java的运行环境，所以可以只安装JRE，不过这里我依然使用JDK，请自行搜索安装。
#rpm -ivh jdk-8u77-linux-x64.rpm
Preparing... ########################################### [100%]
1:jdk1.8.0_77 ########################################### [100%]
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
jfxrt.jar...
#java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
2、安装elasticsearch、logstash、kibana 官网下载地址https://www.elastic.co/downloads/
我使用的版本信息如下：
elasticsearch（下载地址https://www.elastic.co/downloads/past-releases/elasticsearch-2-2-0）
logstash（下载地址https://www.elastic.co/downloads/past-releases/logstash-2-2-0）
kibana （下载地址https://www.elastic.co/downloads/past-releases/kibana-4-4-0）
安装elasticsearch
tar xf elasticsearch-2.2.0.tar.gz -C /usr/local/
安装elasticsearch的head插件
#cd /usr/local/elasticsearch-2.2.0
#./bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Plugins directory [/usr/local/elasticsearch-2.2.0/plugins] does not exist. Creating...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading .................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/local/elasticsearch-2.2.0/plugins/head
查看：
#ll plugins/
total 4
drwxr-xr-x 5 root root 4096 Mar 29 18:09 head
安装elasticsearch的kopf插件
注：Elasticsearch-kopf插件可以查询Elasticsearch中的数据
#./bin/plugin install lmenezes/elasticsearch-kopf
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Downloading ...........................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed kopf into /usr/local/elasticsearch-2.2.0/plugins/kopf
#ll plugins/
total 8
drwxr-xr-x 5 search search 4096 Mar 29 18:09 head
drwxrwxr-x 8 search search 4096 Mar 30 18:10 kopf
创建elasticsearch的data和logs目录
#mkdir /elasticsearch/data -pv
#mkdir /elasticsearch/logs -pv
编辑elasticsearch的配置文件
#cd config
备份一下
#cp elasticsearch.yml elasticsearch.yml_back
#vim elasticsearch.yml --在末尾添加如下几行，
cluster.name: es_cluster
node.name: node-1
path.data: /elasticsearch/data
path.logs: /elasticsearch/logs
network.host: 10.0.90.24
network.port: 9200
启动elasticsearch
#./bin/elasticsearch
Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
提示不能以root用户启动，所以创建一个普通用户，以普通用户身份启动elasticsearch
#groupadd search
#useradd -g search search
将data和logs目录的属主和属组改为search
#chown search.search /elasticsearch/ -R
重新启动
#./bin/elasticsearch
[2016-03-29 19:58:20,026][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/elasticsearch-2.2.0/config/scripts)
Likely root cause: java.nio.file.AccessDeniedException: /usr/local/elasticsearch-2.2.0/config/scripts
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:337)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:248)
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:212)
at org.elasticsearch.bootstrap.Security.configure(Security.java:118)
at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:196)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
报以上错误，原因是权限的问题，修改权限
#chown search.search /usr/local/elasticsearch-2.2.0 -R
然后切换到search用户启动elasticsearch
#su - search
$cd /usr/local/elasticsearch-2.2.0
$./bin/elasticsearch
[2016-03-29 20:11:20,243][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
[2016-03-29 20:11:20,409][INFO ][node ] [node-1] version[2.2.0], pid[2359], build[8ff36d1/2016-01-27T13:32:39Z]
[2016-03-29 20:11:20,409][INFO ][node ] [node-1] initializing ...
[2016-03-29 20:11:21,102][INFO ][plugins ] [node-1] modules [lang-expression, lang-groovy], plugins [head], sites [head]
[2016-03-29 20:11:21,118][INFO ][env ] [node-1] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [24.5gb], net total_space [27.2gb], spins? [possibly], types [ext4]
[2016-03-29 20:11:21,118][INFO ][env ] [node-1] heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-03-29 20:11:22,541][INFO ][node ] [node-1] initialized
[2016-03-29 20:11:22,542][INFO ][node ] [node-1] starting ...
[2016-03-29 20:11:22,616][INFO ][transport ] [node-1] publish_address {10.0.90.24:9300}, bound_addresses {10.0.90.24:9300}
[2016-03-29 20:11:22,636][INFO ][discovery ] [node-1] es_cluster/yNJhglX4RF-ydC4CWpFyTA
[2016-03-29 20:11:25,732][INFO ][cluster.service ] [node-1] new_master {node-1}{yNJhglX4RF-ydC4CWpFyTA}{10.0.90.24}{10.0.90.24:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-03-29 20:11:25,769][INFO ][http ] [node-1] publish_address {10.0.90.24:9200}, bound_addresses {10.0.90.24:9200}
[2016-03-29 20:11:25,770][INFO ][node ] [node-1] started
[2016-03-29 20:11:25,788][INFO ][gateway ] [node-1] recovered [0] indices into cluster_state
也可以直接让elasticsearch在后台运行
$./bin/elasticsearch &
或者不中断启动（我这里使用这种方式启动）
$nohup /usr/local/elasticsearch-2.2.0/bin/elasticsearch &
查看启动是否成功
# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 950/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1027/master
tcp 0 0 ::ffff:10.0.90.24:9300 :::* LISTEN 2428/java
tcp 0 0 :::22 :::* LISTEN 950/sshd
tcp 0 0 ::1:25 :::* LISTEN 1027/master
tcp 0 0 ::ffff:10.0.90.24:9200 :::* LISTEN 2428/java
在浏览器启动查看
http://10.0.90.24:9200/ --会显示如下：
{
"name" : "node-1",
"cluster_name" : "es_cluster",
"version" : {
"number" : "2.2.0",
"build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
"build_timestamp" : "2016-01-27T13:32:39Z",
"build_snapshot" : false,
"lucene_version" : "5.4.1"
},
"tagline" : "You Know, for Search"
}
返回的信息展示了配置的cluster_name和name，以及安装的ES的版本等信息。
注:刚刚安装的head插件，它是一个用浏览器跟ES集群交互的插件，可以查看集群状态、集群的doc内容、执行搜索和普通的Rest请求等。
现在也可以使用它打开http://ip:9200/_plugin/head页面来查看ES集群状态：
http://10.0.90.24:9200/_plugin/head/
可以看到，现在的ES集群中没有index，也没有type，因此这两条是空的。
3、安装kibana
#tar xf kibana-4.4.0-linux-x64.tar.gz -C /usr/local/
#cd /usr/local/
#mv kibana-4.4.0-linux-x64/ kibana
为kibana提供SysV形式的启动脚本
#vi /etc/init.d/kibana
#!/bin/bash
### BEGIN INIT INFO
# Provides: kibana
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Runs kibana daemon
# Description: Runs the kibana daemon as a non-root user
### END INIT INFO
# Process name
NAME=kibana
DESC="Kibana4"
PROG="/etc/init.d/kibana"
# Configure location of Kibana bin
KIBANA_BIN=/usr/local/kibana/bin
# PID Info
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/$NAME
# Configure User to run daemon process
DAEMON_USER=root
# Configure logging location
KIBANA_LOG=/var/log/kibana.log
# Begin Script
RETVAL=0
if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi
# Function library
. /etc/init.d/functions
start() {
echo -n "Starting $DESC : "
pid=`pidofproc -p $PID_FILE kibana`
if [ -n "$pid" ] ; then
echo "Already running."
exit 0
else
# Start Daemon
if [ ! -d "$PID_FOLDER" ] ; then
mkdir $PID_FOLDER
fi
daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &
sleep 2
pidofproc node > $PID_FILE
RETVAL=$?
[[ $? -eq 0 ]] && success || failure
echo
[ $RETVAL = 0 ] && touch $LOCK_FILE
return $RETVAL
fi
}
reload()
{
echo "Reload command is not implemented for this service."
return $RETVAL
}
stop() {
echo -n "Stopping $DESC : "
killproc -p $PID_FILE $DAEMON
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p $PID_FILE $DAEMON
RETVAL=$?
;;
restart)
stop
start
;;
reload)
reload
;;
*)
# Invalid Arguments, print the following message.
echo "Usage: $0 {start|stop|status|restart}" >&2
exit 2
;;
esac
添加执行权限
#chmod +x /etc/init.d/kibana
启动kibana
#service kibana start
Starting Kibana4 : [ OK ]
查看是否启动成功，如下表示启动成功
#netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 950/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1027/master
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 2909/node --kibana端口
tcp 0 0 ::ffff:10.0.90.24:9300 :::* LISTEN 2428/java
tcp 0 0 :::22 :::* LISTEN 950/sshd
tcp 0 0 ::1:25 :::* LISTEN 1027/master
tcp 0 0 ::ffff:10.0.90.24:9200 :::* LISTEN 2428/java
设置开机自启动
#chkconfig --add kibana
#chkconfig kibana on
4、安装Logstash
注：其实它就是一个收集器而已，我们需要为它指定Input和Output（当然Input和Output可以为多个）。
#tar xf logstash-2.2.0.tar.gz -C /usr/local/
#cd /usr/local/
#mv logstash-2.2.0 logstash
测试logstash
#/usr/local/logstash/bin/logstash -e 'input { stdin{} } output { stdout {} }' --你会发现输入什么内容，logstash按照某种格式输出什么内容
Settings: Default pipeline workers: 2
Logstash startup completed
hello world ---输入的内容
2016-04-01T09:05:35.818Z elk hello world
注：其中-e参数允许Logstash直接通过命令行接受设置。这点尤其快速的帮助我们反复的测试配置是否正确而不用写配置文件。使用CTRL-C命令可以退出之前运行的Logstash。
使用-e参数在命令行中指定配置是很常用的方式，不过如果需要配置更多设置则需要很长的内容。这种情况，我们首先创建一个简单的配置文件，并且指定logstash使用这个配置文件。
注：logstash 配置文件的例子：https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
logstash配置文件是以json格式设置参数的配置文件位于/etc/logstash/conf.d目录（rpm安装的路径）下配置包括三个部分输入端、过滤器和输出。
格式如下：
# This is a comment. You should use comments to describe
# parts of your configuration.
input {
...
}
filter {
...
}
output {
...
}
插件配置格式：
input {
file {
path => "/var/log/messages"
type => "syslog"
}
file {
path => "/var/log/apache/access.log"
type => "apache"
}
}
首先创建一个简单的例子
#cd /usr/local/logstash/config
#cat logstash-simple.conf
input { stdin { } }
output {
stdout { codec => rubydebug }
}
执行：
先输出一些内容，例如当前时间：
#echo "`date` hello world"
Fri Apr 1 17:07:17 CST 2016 hello world
#/usr/local/logstash/bin/logstash agent -f logstash-simple.conf --执行
Settings: Default pipeline workers: 2
Logstash startup completed
Fri Apr 1 17:07:17 CST 2016 hello world --将刚才生成的时间信息粘贴到这里，回车，就会看到如下信息：
{
"message" => "Tue Jul 14 18:07:07 EDT 2015 hello World",
"@version" => "1",
"@timestamp" => "2016-04-01T09:08:19.809Z",
"host" => "elk"
}
接下来在logstash的安装目录创建一个用于测试logstash使用elasticsearch作为logstash的后端输出的测试文件
logstash-es-test.conf该文件中定义了stdout和elasticsearch作为output，这样的“多重输出”即保证输出结果显示到屏幕上，同时也输出到elastisearch中。如下：
#cat logstash-es-test.conf
input { stdin { } }
output {
elasticsearch {hosts => "10.0.90.24" }
stdout { codec=> rubydebug }
}
测试配置文件是否正确
/usr/local/logstash/bin/logstash --configtest -f logstash-es-test.conf
Configuration OK
如果文件比较多也可以这样：
/usr/local/logstash/bin/logstash --configtest -f config/*.conf
执行：
/usr/local/logstash/bin/logstash agent -f logstash-es-test.conf
Settings: Default pipeline workers: 2
Logstash startup completed
hello logstash --输入内容，回车
{
"message" => "hello logstash",
"@version" => "1",
"@timestamp" => "2016-04-01T09:18:26.967Z",
"host" => "elk"
}
Ctrl+c 结束执行！
我们可以使用curl命令发送请求来查看ES是否接收到了数据：
#curl 'http://10.0.90.24:9200/_search?pretty'
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 6,
"successful" : 6,
"failed" : 0
},
"hits" : {
"total" : 5,
"max_score" : 1.0,
"hits" : [ {
"_index" : ".kibana",
"_type" : "config",
"_id" : "4.4.0",
"_score" : 1.0,
"_source" : {
"buildNum" : 9689
}
}, {
"_index" : "logstash-2016.04.01",
"_type" : "logs",
"_id" : "AVPRHddUspScKx_yDLKx",
"_score" : 1.0,
"_source" : {
"message" : "hello logstash",
"@version" : "1",
"@timestamp" : "2016-04-01T09:18:26.967Z",
"host" : "elk"
}
}]
}
}
通过以上显示信息，可以看到ES已经收到了数据！说明可以通过Elasticsearch和Logstash来收集日志数据了。
5、修改kibana端口
#cd /usr/local/kibana/config
备份配置
#cp kibana.yml kibana.yml_back
修改为如下：其他默认不变
server.port: 80 --修改端口为80，默认是5601
server.host: "10.0.90.24"
elasticsearch.url: "http://10.0.90.24:9200" --ip为server的ip地址
kibana.defaultAppId: "discover"
elasticsearch.requestTimeout: 300000
elasticsearch.shardTimeout: 0
重启kibana
#service kibana restart
在浏览器访问kibana：
http://10.0.90.24 --就可以看到kibana的页面了
登录之后，首先配置一个索引，默认kibana的数据被指向Elasticsearch，使用默认的logstash-*的索引名称，并且是基于时间（@timestamp）的，如下
点击“Create”，看到如下界面说明索引创建完成。
点击“Discover”，可以搜索和浏览Elasticsearch中的数据，默认搜索的是最近15分钟的数据，可以自定义选择时间。
到此，说明你的ELK平台安装部署完成。
6、配置logstash作为Indexer
将logstash配置为索引器，并将logstash的日志数据存储到Elasticsearch，本范例主要是索引本地系统日志
#cd /usr/local/logstash/config
#cat logstash-indexer.conf
input {
file {
type => "syslog"
path => ["/var/log/messages", "/var/log/secure" ]
}
syslog {
type => "syslog"
port => "5544"
}
}
output {
elasticsearch { hosts => "10.0.90.24" }
stdout { codec => rubydebug }
}
测试是否有语法错误：
#/usr/local/logstash/bin/logstash --configtest -f logstash-indexer.conf
Configuration OK
启动
nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash-indexer.conf &
查看端口：
# netstat -tunlp
 然后到kibana界面刷新下，就可以看到日志信息了。
测试echo一条日志信息到/var/log/messages,然后再通过kibana界面查看
#echo "`date` This is a test for logstash for indexer" >> /var/log/messages
如下图：
测试从一台服务器（ip为10.0.18.12）登录到10.0.90.24
#ssh root@10.0.90.24
The authenticity of host '10.0.90.24 (10.0.90.24)' can't be established.
RSA key fingerprint is 4b:97:0a:97:e8:cf:a5:39:49:6c:65:8e:32:79:64:c8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.90.24' (RSA) to the list of known hosts.
root@10.0.90.24's password: --输入10.0.90.24 服务器的root密码
Last login: Fri Apr 1 17:31:32 2016 from 10.0.90.8
然后查看kibana，看是否搜集到了日志
 以上表示搜集日志成功。
三、在客户端安装配置（即需要收集日志的服务器）
注：客户端IP是10.0.90.25，安装配置了httpd服务
1、安装jdk
# rpm -ivh jdk-8u77-linux-x64.rpm
Preparing... ########################################### [100%]
1:jdk1.8.0_77 ########################################### [100%]
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
jfxrt.jar...
测试jdk安装是否成功
#java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
表示安装OK
2、安装logstash
#tar xf logstash-2.2.0.tar.gz -C /usr/local/
#cd /usr/local/
#mv logstash-2.2.0 logstash
创建配置文件
#cd logstash
#mkdir config
#vim logstash-http.conf --内容如下
input {
file {
path => "/var/log/httpd/access_log"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["10.0.90.24:9200"] --ELK服务端ip地址和端口
index => "http-access-log-%{+YYYY.MM.dd.HH}" --index名称，自定义的
workers =>5
template_overwrite => true
}
}
表示将httpd的access日志output到ES上，并通过kibana显示日志。
检测配置文件是否有语法错误！OK表示没有错误
#/usr/local/logstash/bin/logstash --configtest -f logstash-http.conf
Configuration OK
3、启动logstash
#nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash-http.conf &
查看进程，如下图
 到kibana界面创建一个新的index
点击“Settings”,填写如下内容
然后点击“Create”，出现如下图界面，表示index创建成功
在客户端测试日志output到ES情况：在access_log日志中echo一行测试日志，如下：
#echo "This is a test line for http log" >> /var/log/httpd/access_log
然后在浏览器访问10.0.90.25服务器的http服务，http://10.0.90.25
在kibana界面查看，可以看到如下图
参考地址：
http://baidu.blog.运维网.com/71938/1676798
注：ELK刚开始研究，如有不足之处请多多指出，谢谢！
ELKstack中文指南：http://kibana.logstash.es/content/logstash/get_start/index.html
|
|
QQ群⑧：
窥视卡
雷达卡
发表于 2019-1-29 12:35:04
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡