设为首页 收藏本站
查看: 1674|回复: 0

[经验分享] Hadoop项目之Cloudera 5.6.0.1(CDH)的安装部署

[复制链接]

尚未签到

发表于 2018-10-29 13:08:26 | 显示全部楼层 |阅读模式
  1 前言
  Cloudera是基于Apache原生的Hadoop组件进行重新封装和加强,Cloudera可以简化Hadoop组件的部署。
  更新版本的配置请参阅(CentOS 7):
  http://cmdschool.blog.51cto.com/2420395/1916322
  2 理论基础
  2.1 部署软件架构
  1)Oracle JDK
  2)Cloudera Manager Server and Agent packages
  3)Supporting database software
  4)CDH and managed service software
  2.2 部署步骤和安装方法
  2.2.1 安装方法
  A)Cloudera Manager安装程序安装(容易)
  B)yum源方式安装(中等)
  C)源代码安装(难)
  注:本教程使用方法B
  2.2.2 部署步骤
  1)安装JDK
  2)安装并配置数据库
  3)安装Cloudera管理服务端
  4)安装Cloudera管理代理端
  5)安装CDH管理服务软件
  6)创建、启动和配置CDH并管理服务
  2.3 Cloudera Manager端的相关文件
rpm -ql cloudera-manager-server  显示如下:
/etc/cloudera-scm-server  
/etc/cloudera-scm-server/db.properties
  
/etc/cloudera-scm-server/log4j.properties
  
/etc/default/cloudera-scm-server
  
/etc/rc.d/init.d/cloudera-scm-server
  
/opt/cloudera/csd
  
/opt/cloudera/parcel-repo
  
/usr/sbin/cmf-server
  
/var/log/cloudera-scm-server
  
/var/run/cloudera-scm-server
  文件与目录功能如下:
  1)其中/etc/的2-4行为Cloudera Manager服务端配置文件
  2)/opt/cloudera/parcel-repo为下载安装包存放目录
  3 实践部分
  3.1 环境信息
  3.1.1 系统信息
  OS = CentOS 6.6 x86_64
  注:系统请使用最小化安装,否则可能Sqoop服务可能无法启动
  3.1.2 主机信息
  Cloudera Manager:
  ip address=10.168.0.120
  hostname=cdm-m.cmdschool.org
  Cloudera Host1:
  ip address=10.168.0.121
  hostname=cdm-h1.cmdschool.org
  Cloudera Host2:
  ip address=10.168.0.122
  hostname=cdm-h2.cmdschool.org
  Cloudera Host3:
  ip address=10.168.0.123
  hostname=cdm-h3.cmdschool.org
  Cloudera Host4:
  ip address=10.168.0.124
  hostname=cdm-h4.cmdschool.org
  3.2 运行环境配置
  In Cloudera Manager & Cloudera Host[1-4]
  3.2.1 关闭selinux
getenforce  如果显示如下:
Enforcing  则执行:
setenforce 0  
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
  3.2.2 配置hosts
  vim编辑/etc/hosts
10.168.0.120 cdm-m.cmdschool.org  
10.168.0.121 cdm-h1.cmdschool.org
  
10.168.0.122 cdm-h2.cmdschool.org
  
10.168.0.123 cdm-h3.cmdschool.org
  
10.168.0.124 cdm-h4.cmdschool.org
  3.2.3 检查主机名称
hostname  注:主机名称与上面不一致会影响服务的启动
  3.2.4 配置sudo(单用户模式适用,可选)
visudo  增加如下组
%cloudera-scm ALL=(ALL) NOPASSWD: ALL  确认包含如下行:
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin  vim编辑/etc/pam.d/su,确保包含如下行
session         required        pam_limits.so  3.2.5 关闭防火墙并设置开机不启动
/etc/init.d/iptables stop  
chkconfig iptables off
  
  3.2.6 优化虚拟内存需求率
  1)检查虚拟内存需求率
cat /proc/sys/vm/swappiness  显示如下:
60  2)临时降低虚拟内存需求率
sysctl vm.swappiness=0  3)永久降低虚拟内存需求率
  vim编辑/etc/sysctl.conf
kernel.shmall = 4294967296  
vm.swappiness = 0
  并运行如下命令使生效
sysctl -p  3.2.7 解决透明大页面问题
  1)检查透明大页面问题
cat /sys/kernel/mm/transparent_hugepage/defrag  如果显示为:
[always] madvise never  2)临时关闭透明大页面问题
echo never > /sys/kernel/mm/transparent_hugepage/defrag  确认配置生效:
cat /sys/kernel/mm/transparent_hugepage/defrag  应该显示为:
always madvise [never]  3)配置开机自动生效
  vim编辑/etc/rc.local,加入如下内容
echo never > /sys/kernel/mm/transparent_hugepage/defrag  3.3 yum源的安装配置
  3.3.1 公共yum源配置
  In Cloudera Manager & Cloudera Host[1-4]
  1)配置yum源
  下载默认yum源
wget -P /etc/yum.repos.d/ https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo  修改为指定版本yum源
  vim编辑/etc/yum.repos.d/cloudera-manager.repo修改如下参数:
baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.6.0/  2)安装配置工具
yum install -y vim wget openssh-clients  3)安装jdk
yum install -y oracle-j2sdk1.7  4)安装python
yum install -y python  
  5)安装ntpd
yum install -y ntp  3.3.2 Cloudera Manager端yum源配置
  In Cloudera Manager
  1)安装Cloudera Manager包
yum install -y cloudera-manager-daemons cloudera-manager-server  2)安装mysql
yum install -y mysql-server mysql-devel mysql  3.3.3 Cloudera Manager Agent端yum源配置
  In Cloudera Host[1-4]
  安装Cloudera Manager Agent包
yum install -y cloudera-manager-agent cloudera-manager-daemons  3.4 依赖于yum源的环境配置
  3.4.1 配置jdk变量环境
  In Cloudera Manager & Cloudera Host[1-4]
  1)vim编辑/etc/profile,末尾加入如下内容
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera  
export JRE_HOME=${JAVA_HOME}/jre
  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
  
export PATH=${JAVA_HOME}/bin:$PATH
  2)导入java环境变量
source /etc/profile  3)测试jdk的配置
java -version  3.4.2 权限检查(单用户模式适用,可选)
  In Cloudera Manager & Cloudera Host[1-4]
  检查以下目录cloudera-scm用户具有完全的权限
  检查当前目录权限:
ls -ld /opt/cloudera/  显示如下
drwxr-xr-x. 4 cloudera-scm cloudera-scm 4096 May 23 13:51 /opt/cloudera/  检查子目录权限:
ls -lR /opt/cloudera/  显示如下
/opt/cloudera/:  
total 8
  
drwxr-xr-x. 2 cloudera-scm cloudera-scm 4096 Feb 12 11:28 csd
  
drwxr-xr-x. 2 cloudera-scm cloudera-scm 4096 Feb 12 11:28 parcel-repo
  

  
/opt/cloudera/csd:
  
total 0
  

  
/opt/cloudera/parcel-repo:
  
total 0
  同样,检查服务器或客户端目录权限
ls -ld /var/log/cloudera-scm-server/  
ls -lR /var/log/cloudera-scm-server/
  
ls -ld /var/lib/cloudera-scm-agent/
  
ls -lR /var/lib/cloudera-scm-agent/
  3.4.3 检查线程限制配置
  In Cloudera Manager & Cloudera Host[1-4]
cat /etc/security/limits.d/cloudera-scm.conf  显示如下:
#  
# (c) Copyright 2014 Cloudera, Inc.
  
#
  
cloudera-scm    soft  nofile  32768
  
cloudera-scm    soft  nproc   65536
  
cloudera-scm    hard  nofile  1048576
  
cloudera-scm    hard  nproc   unlimited
  
cloudera-scm    hard  memlock unlimited
  
  3.4.4 Cloudera Manager端配置
  In Cloudera Manager
  1)临时校对时间
ntpdate 0.centos.pool.ntp.org  2)启动并配置ntpd服务自动启动
/etc/init.d/ntpd start  
chkconfig ntpd on
  3.4.5 Cloudera Manager Agen端配置
  In Cloudera Host[1-4]
  1)临时校对时间
ntpdate 10.168.0.120  2)vim编辑/etc/ntp.conf
  注释掉外网时间服务器并增加内网时间服务器地址
#server 0.centos.pool.ntp.org iburst  
#server 1.centos.pool.ntp.org iburst
  
#server 2.centos.pool.ntp.org iburst
  
#server 3.centos.pool.ntp.org iburst
  
server 10.168.0.120 iburst
  3)启动并配置ntpd服务自动启动
/etc/init.d/ntpd start  
chkconfig ntpd on
  
  3.4.7 安装MySQL JDBC Driver
  In Cloudera Manager & Cloudera Host[1-4]
wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.39.tar.gz  
tar zxvf mysql-connector-java-5.1.39.tar.gz
  
mkdir /usr/share/java/
  
cp mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar /usr/share/java/mysql-connector-java.jar
  
  3.4.8 配置公钥认证
  In Cloudera Manager
  In Cloudera Manager:
ssh-keygen -t rsa  注:以上一路回车
  In Cloudera Manager Agen:
ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.168.0.120  
ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.168.0.121
  
ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.168.0.122
  
ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.168.0.123
  
ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.168.0.124
  In Cloudera Manager:
ssh 10.168.0.120  
ssh 10.168.0.121
  
ssh 10.168.0.122
  
ssh 10.168.0.123
  
ssh 10.168.0.124
  注:以上如果无需密码登记即成功
  3.5 Cloudera Manager安装配置
  In Cloudera Manager
  3.5.1 修改mysql参数
  1)关闭数据库
/etc/init.d/mysqld stop  2)备份ib_logfile文件
mkdir /var/lib/backup  
cd /var/lib/mysql/
  
mv ib_logfile* /var/lib/backup/
  3)vim编辑/etc/my.cnf
  加入如下参数
[mysqld]  
transaction-isolation = READ-COMMITTED
  
# Disabling symbolic-links is recommended to prevent assorted security risks;
  
# to do so, uncomment this line:
  
# symbolic-links = 0
  

  
key_buffer = 16M
  
key_buffer_size = 32M
  
max_allowed_packet = 32M
  
thread_stack = 256K
  
thread_cache_size = 64
  
query_cache_limit = 8M
  
query_cache_size = 64M
  
query_cache_type = 1
  

  
max_connections = 550
  
#expire_logs_days = 10
  
#max_binlog_size = 100M
  

  
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
  
#and chown the specified folder to the mysql user.
  
log_bin=/var/lib/mysql/mysql_binary_log
  

  
# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
  
binlog_format = mixed
  

  
read_buffer_size = 2M
  
read_rnd_buffer_size = 16M
  
sort_buffer_size = 8M
  
join_buffer_size = 8M
  

  
# InnoDB settings
  
innodb_file_per_table = 1
  
innodb_flush_log_at_trx_commit  = 2
  
innodb_log_buffer_size = 64M
  
innodb_buffer_pool_size = 4G
  
innodb_thread_concurrency = 8
  
innodb_flush_method = O_DIRECT
  
innodb_log_file_size = 512M
  

  
[mysqld_safe]
  
log-error=/var/log/mysqld.log
  
pid-file=/var/run/mysqld/mysqld.pid
  

  
sql_mode=STRICT_ALL_TABLES
  3.5.2 启动并设置开机自动启动
/etc/init.d/mysqld start  
chkconfig mysqld on
  3.5.3 初始化数据库
mysql_secure_installation  向导如下:
[...]  
Enter current password for root (enter for none):
  
OK, successfully used password, moving on...
  
[...]
  
Set root password? [Y/n] y
  
New password:
  
Re-enter new password:
  
Remove anonymous users? [Y/n] y
  
[...]
  
Disallow root login remotely? [Y/n] n
  
[...]
  
Remove test database and access to it [Y/n] y
  
[...]
  
Reload privilege tables now? [Y/n] y
  
All done!
  3.5.4 准备scm库
  1)方法一
  数据库配置:
mysql -uroot -p  
create database scm default character set utf8;
  
grant all privileges on *.* to scm@'cdm-m.cmdschool.org' identified by 'scm';
  
flush privileges;
  vim编辑/etc/cloudera-scm-server/db.properties修改如下参数:
com.cloudera.cmf.db.type=mysql  
com.cloudera.cmf.db.host=cdm-m.cmdschool.org
  
com.cloudera.cmf.db.name=scm
  
com.cloudera.cmf.db.user=scm
  
com.cloudera.cmf.db.password=scm
  2)方法二(官方建议)
  授权temp权限:
mysql -uroot -p  
grant all privileges on *.* to 'temp'@'%' identified by 'temp' with grant option;
  
flush privileges;
  生成配置文件:
/usr/share/cmf/schema/scm_prepare_database.sh mysql -h cdm-m.cmdschool.org -utemp -ptemp --scm-host cdm-m.cmdschool.org scm scm scm  显示如下:
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera  
Verifying that we can write to /etc/cloudera-scm-server
  
Creating SCM configuration file in /etc/cloudera-scm-server
  
Executing: /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
  
[main] DbCommandExecutor INFO Successfully connected to database.
  
All done, your SCM database is configured correctly!
  确认生成的结果:
cat /etc/cloudera-scm-server/db.properties  显示如下:
# Auto-generated by scm_prepare_database.sh on Tue May 24 19:08:19 CST 2016  
#
  
# For information describing how to configure the Cloudera Manager Server
  
# to connect to databases, see the "Cloudera Manager Installation Guide."
  
#
  
com.cloudera.cmf.db.type=mysql
  
com.cloudera.cmf.db.host=cdm-m.cmdschool.org
  
com.cloudera.cmf.db.name=scm
  
com.cloudera.cmf.db.user=scm
  
com.cloudera.cmf.db.password=scm
  确认库访问权限:
mysql -uroot -p  
show grants for scm@'cdm-m.cmdschool.org';
  显示如下:
+----------------------------------------------------------------------------------------------------------------------+  
| Grants for scm@cdm-m.cmdschool.org                                                                                   |
  
+----------------------------------------------------------------------------------------------------------------------+
  
| GRANT USAGE ON *.* TO 'scm'@'cdm-m.cmdschool.org' IDENTIFIED BY PASSWORD '*45E6E3C68BDF1AC7EBB5C5A3BCBD5E9437B293BE' |
  
| GRANT ALL PRIVILEGES ON `scm`.* TO 'scm'@'cdm-m.cmdschool.org'                                                       |
  
+----------------------------------------------------------------------------------------------------------------------+
  
2 rows in set (0.00 sec)
  清理数据库用户授权:
drop user 'temp'@'%';  
flush privileges;
  3.5.5 创建附加数据库(可选)
  1)附加数据库列表
RoleDatabaseUserPasswordActivity Monitoramonamonamon_passwordReports Managerrmanrmanrman_passwordHive Metastore Servermetastorehivehive_passwordSentry Serversentrysentrysentry_passwordCloudera Navigator Audit Servernavnavnav_password  Cloudera Navigator Metadata Server
navmsnavmsnavms_password  2)创建数据库并配置管理账号密码
mysql -uroot -p  
create database amon default character set utf8;
  
grant all privileges on amon.* to 'amon'@'%' identified by 'amon_password';
  

  
create database rman default character set utf8;
  
grant all privileges on rman.* to 'rman'@'%' identified by 'rman_password';
  

  
create database metastore default character set utf8;
  
grant all privileges on metastore.* to 'hive'@'%' identified by 'hive_password';
  

  
create database sentry default character set utf8;
  
grant all privileges on sentry.* to 'sentry'@'%' identified by 'sentry_password';
  

  
create database nav default character set utf8;
  
grant all privileges on nav.* to 'nav'@'%' identified by 'nav_password';
  

  
create database navms default character set utf8;
  
grant all privileges on navms.* to 'navms'@'%' identified by 'navms_password';
  

  
flush privileges;
  3.5.6 配置Oozie库(可选)
  1)数据库权限配置
mysql -uroot -p  
create database oozie default character set utf8;
  
grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';
  
grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';
  
flush privileges;
  2)配置Oozie库所需软连接
cd /opt/cloudera/parcels/CDH/lib/oozie/lib/  
ln -s /usr/share/java/mysql-connector-java.jar mysql-connector-java.jar
  3.5.7 启动服务并配置开机启动
/etc/init.d/cloudera-scm-server start  
chkconfig cloudera-scm-server on
  3.5.8 故障排除
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.out  3.6 Cloudera Manager Agent安装
  In Cloudera Host[1-4]
  3.6.1 创建压缩包存放目录
mkdir -p /opt/cloudera/parcels  
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
  3.6.2 指定管理服务器和指定包存放目录
  vim编辑/etc/cloudera-scm-agent/config.ini确保参数如下并启用:
server_host=cdm-m.cmdschool.org  
server_port=7182
  
parcel_dir=/opt/cloudera/parcels
  3.6.3 指定运行单用户模式的用户名(仅用于单用户模式,不配置)
  vim编辑/etc/default/cloudera-scm-agent并取消以下行的注释
USER="cloudera-scm"  3.6.4 启动服务并配置服务器开机启动
/etc/init.d/cloudera-scm-agent start  
chkconfig cloudera-scm-agent on
  3.6.5 故障排除
  如下命令监控启动服务的错误输出
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.out  3.7 登陆配置
  
  In Cloudera WEB Manager
  http://10.168.0.120:7180/
DSC0000.png

  界面配置部分本章节省略……
  
  更新版本的配置请参阅(CentOS 7):
  http://cmdschool.blog.51cto.com/2420395/1916322
  ---------------------------------------
  官方文档
  http://www.cloudera.com/downloads/manager/5-6-0.html
  中文文档
  http://www.cloudera.com/content/www/zh-CN/documentation/enterprise/5-3-x/topics/installation_installation.html
  部署方法B的目录
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_install_path_b.html
  单用户模式配置
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/install_singleuser_reqts.html#xd_583c10bfdbd326ba--69adf108-1492ec0ce48--7ade__section_nhj_dlh_kq
  mysql的配置概述
  www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_installing_configuring_dbs.html
  MySQL&MySQL JDBC Driver的详细安装配置
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_mysql.html#cmig_topic_5_5
  MySQL JDBC Driver的下载地址
  http://dev.mysql.com/downloads/connector/j/5.1.html
  JDK的安装配置
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_jdk_installation.html
  配置SCM库
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_installing_configuring_dbs.html#concept_i2r_m3m_hn__section_qjj_pyp_bm
  配置Oozie库
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/install_oozie_ext_db.html#admin_oozie_ext_db
  Hue服务的数据库配置
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_mc_hue_service.html#xd_583c10bfdbd326ba--7f25092b-13fba2465e5--7e7c
  Agent的配置
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ag_agent_config.html#cmug_topic_5_16__section_kw3_5rq_wm
  CDH服务的安装
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_path_b_manual.html
  命令行停止CDH服务
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_services_stop.html#topic_27_3
  各版的yum源(下载)地址
  http://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_vd.html
  CDH的yum源下载地址
  http://archive.cloudera.com/cdh5/one-click-install/redhat/6/x86_64/
  CDH的手动安装
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_path_b_manual.html
  CDH包下载
  http://archive.cloudera.com/cdh5/parcels/
  http://archive.cloudera.com/gplextras5/parcels/
  CDH包的下载地址:
  http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_create_local_parcel_repo.html#cmig_topic_21_5
  非官方的文档
  http://www.it165.net/database/html/201604/15043.html
  http://wenku.baidu.com/link?url=x9yTezHZKK2a707ZLN6I8rPkdQbzBAuyy4HOzvPSJqvNJPTZmHnsvhalsEJxYPtbJD2Xybi9tnI0HhjTlrkds27c_-Se4MWfX6v45lzg9VO
  http://www.tuicool.com/articles/QFZV3ij
  http://www.tuicool.com/articles/ENjmeaY/
  进程限制配置原理
  http://www.iyunv.com/os/201309/240804.html



运维网声明 1、欢迎大家加入本站运维交流群:群②:261659950 群⑤:202807635 群⑦870801961 群⑧679858003
2、本站所有主题由该帖子作者发表,该帖子作者与运维网享有帖子相关版权
3、所有作品的著作权均归原作者享有,请您和我们一样尊重他人的著作权等合法权益。如果您对作品感到满意,请购买正版
4、禁止制作、复制、发布和传播具有反动、淫秽、色情、暴力、凶杀等内容的信息,一经发现立即删除。若您因此触犯法律,一切后果自负,我们对此不承担任何责任
5、所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其内容的准确性、可靠性、正当性、安全性、合法性等负责,亦不承担任何法律责任
6、所有作品仅供您个人学习、研究或欣赏,不得用于商业或者其他用途,否则,一切后果均由您自己承担,我们对此不承担任何法律责任
7、如涉及侵犯版权等问题,请您及时通知我们,我们将立即采取措施予以解决
8、联系人Email:admin@iyunv.com 网址:www.yunweiku.com

所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其承担任何法律责任,如涉及侵犯版权等问题,请您及时通知我们,我们将立即处理,联系人Email:kefu@iyunv.com,QQ:1061981298 本贴地址:https://www.yunweiku.com/thread-628043-1-1.html 上篇帖子: 【科普】一篇文让你了解Hadoop-11611907 下篇帖子: Hadoop Block-HBASE
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

扫码加入运维网微信交流群X

扫码加入运维网微信交流群

扫描二维码加入运维网微信交流群,最新一手资源尽在官方微信交流群!快快加入我们吧...

扫描微信二维码查看详情

客服E-mail:kefu@iyunv.com 客服QQ:1061981298


QQ群⑦:运维网交流群⑦ QQ群⑧:运维网交流群⑧ k8s群:运维网kubernetes交流群


提醒:禁止发布任何违反国家法律、法规的言论与图片等内容;本站内容均来自个人观点与网络等信息,非本站认同之观点.


本站大部分资源是网友从网上搜集分享而来,其版权均归原作者及其网站所有,我们尊重他人的合法权益,如有内容侵犯您的合法权益,请及时与我们联系进行核实删除!



合作伙伴: 青云cloud

快速回复 返回顶部 返回列表