mongodb集群Replica Set +Sharding高可用集群搭建(含认证)
一、概述1、架构
2、唠叨概念
Replica Set
中文翻译：副本集
简单的说集群中包含了多份数据，保证主节点挂掉了，备用节点继续提供数据服务，提供的前提就是数据需要主节点一致
仲裁节点是一种特殊的节点，它本身不存储数据，主要的作用是决定哪一个备用节点在主节点挂掉之后提升为主节点，所以客户端不需要连接此节点。
这里虽然只有备节点，但是仍然需要一个仲裁节点来提升节点级别。我开始也不相信必须要有仲裁节点，但是自已也试过如果没有仲裁节点的话
主节点挂掉后，备节点还是备节点，所以咱们还需要它的。
主节点和备用节点
主备节点存储数据，仲裁节点不存储数据
默认情况：主节点提供增删查改功能，备节点不提供任何服务。但是可以设置备节点提供查询功能，这样就可以减少主节点的压力，当客户端进行数据查询时
请求自动转到备用节点上，这个设置叫做Read Preference Modes，同时Java客户端提供了简单的配置方式，可以不必直接对数据库操作。
二、地址和服务规划
192.168.0.178 master mongos:20000 config:21000 shard1:22001 shard2:22002 shard3:22003
192.168.0.180 slave1 mongos:20000 config:21000 shard1:22001 shard2:22002 shard3:22003
192.168.0.181 slave2 mongos:20000 config:21000 shard1:22001 shard2:22002 shard3:22003
三、版本介绍
操作系统版本：CentOS 6.4 64bit
mongodb版本：2.6.1 二进制通用包
四、准备工作
1、关闭iptables和selinux防火墙
2、主机名互为解析(可参考，非必须)
3、服务器之间时间同步(重点)
4、创建数据和日志存储目录
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
mkdir -p /data/mongodb
cd /data/mongodb
mkdir mongos
mkdir mongos/log
mkdir config
mkdir config/{data,log}
mkdir shard1
mkdir shard1/{data,log}
mkdir shard2
mkdir shard2/{data,log}
mkdir shard3
mkdir shard3/{data,log}
cd ~
五、mongodb安装配置
1、下载mongodb安装包
官网下载地址：http://www.mongodb.org/
2、下载解压mongodb二进制通用包，解压后就能使用
tar xf mongodb-linux-x86_64-2.6.1.tgz -C /usr/local/
cd /usr/local/
ln -sv mongodb-linux-x86_64-2.6.1 mongodb
3、配置mongodb环境变量
cat > /etc/profile.d/mongodb.sh << EOF
export MONGODB_HOME=/usr/local/mongodb
export PATH=\$PATH:\$MONGODB_HOME/bin
EOF
source /etc/profile.d/mongodb.sh
六、启动服务
1、分别在master、slave1和slave2服务器上启动配置服务器
mongod --configsvr --dbpath /data/mongodb/config/data --port 21000 --logpath /data/mongodb/config/log/config.log --fork
2、分别在master、slave1和slave2服务器上启动mongos服务器
mongos--configdb 192.168.0.178:21000,192.168.0.180:21000,192.168.0.181:21000 --port 20000 --logpath/data/mongodb/mongos/log/mongos.log --fork
3、分别在master、slave1和slave2服务器上配置分片副本集
mongod --shardsvr --replSet shard1 --port 22001 --dbpath /data/mongodb/shard1/data--logpath /data/mongodb/shard1/log/shard1.log --fork --nojournal--oplogSize 10
mongod --shardsvr --replSet shard2 --port 22002 --dbpath /data/mongodb/shard2/data--logpath /data/mongodb/shard1/log/shard2.log --fork --nojournal--oplogSize 10
mongod --shardsvr --replSet shard3 --port 22003 --dbpath /data/mongodb/shard3/data--logpath /data/mongodb/shard1/log/shard3.log --fork --nojournal--oplogSize 10
4、登陆任意一台mongodb服务器，比如
###设置第一个分片副本集，必须使用admin数据库
mongo 192.168.0.178:22001/admin
#定义副本集
> config = { _id:"shard1", members:[
{_id:0,host:"192.168.0.178:22001",priority:1},
{_id:1,host:"192.168.0.180:22001",priority:2},
{_id:2,host:"192.168.0.181:22001",arbiterOnly:true}
]
}
#初始化副本集
> rs.initiate(config);
###设置第二个分片副本集，必须使用admin数据库
mongo 192.168.0.180:22002/admin
#定义副本集
> config = { _id:"shard2", members:[
{_id:0,host:"192.168.0.178:22002",arbiterOnly:true},
{_id:1,host:"192.168.0.180:22002",priority:1},
{_id:2,host:"192.168.0.181:22002",priority:1}
]
}
#初始化副本集
> rs.initiate(config);
###设置第三个分片副本集，必须使用admin数据库
mongo 192.168.0.181:22003/admin
#定义副本集
> config = { _id:"shard3", members:[
{_id:0,host:"192.168.0.178:22003",priority:2},
{_id:1,host:"192.168.0.180:22003",arbiterOnly:true},
{_id:2,host:"192.168.0.181:22003",priority:1}
]
}
#初始化副本集
> rs.initiate(config);
5、目前搭建了mongodb配置服务器、路由服务器，各个分片服务器，不过应用程序连接到 mongos 路由服务器并不能使用分片机制，还需要在程序里设置分片配置，让分片生效。
mongo 192.168.0.178:20000/admin
#串联路由服务器与分配副本集1
db.runCommand( { addshard : "shard1/192.168.0.178:22001,192.168.0.180:22001,192.168.0.181:22001"});
#串联路由服务器与分配副本集2
db.runCommand( { addshard : "shard2/192.168.0.178:22002,192.168.0.180:22002,192.168.0.181:22002"});
#串联路由服务器与分配副本集3
db.runCommand( { addshard : "shard3/192.168.0.178:22003,192.168.0.180:22003,192.168.0.181:22003"});
6、查看分片服务器配置
db.runCommand({listshards : 1 });
命令输出结果
mongos> db.runCommand({listshards : 1 });
{
"shards" : [
{
"_id" : "shard1",
"host" : "shard1/192.168.0.178:22001,192.168.0.180:22001"
},
{
"_id" : "shard2",
"host" : "shard2/192.168.0.178:22002,192.168.0.180:22002"
},
{
"_id" : "shard3",
"host" : "shard3/192.168.0.178:22003,192.168.0.180:22003"
}
],
"ok" : 1
}
因为192.168.0.181是每个分片副本集的仲裁节点，所以在上面结果没有列出来。
7、目前配置服务、路由服务、分片服务、副本集服务都已经串联起来了，但我们的目的是希望插入数据，数据能够自动分片，就差那么一点点，一点点。。。
连接在mongos上，准备让指定的数据库、指定的集合分片生效。
#指定uba分片生效
db.runCommand( { enablesharding :"uba"});
#指定数据库里需要分片的集合和片键
db.runCommand( { shardcollection : "uba.table1",key : {id: 1} } )
我们设置uba的 table1 表需要分片，根据 id 自动分片到 shard1 ，shard2，shard3 上面去。要这样设置是因为不是所有mongodb 的数据库和表 都需要分片！
8、登陆mongos添加用户
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# mongo 192.168.0.178:20000/admin
MongoDB shell version: 2.6.1
connecting to: 192.168.0.178:20000/admin
mongos> db
admin
mongos> db.addUser('admin','allentuns')
WARNING: The 'addUser' shell helper is DEPRECATED. Please use 'createUser' instead
Successfully added user: { "user" : "admin", "roles" : [ "root" ] }
mongos> db.addUser('admin_read','allentuns_read'，true)#添加超级管理员可读用户
WARNING: The 'addUser' shell helper is DEPRECATED. Please use 'createUser' instead
Successfully added user: { "user" : "admin_read", "roles" : [ "read" ] }
mongos> use uba
switched to db uba
mongos> db.addUser('sa','123root')
WARNING: The 'addUser' shell helper is DEPRECATED. Please use 'createUser' instead
Successfully added user: { "user" : "sa", "roles" : [ "dbOwner" ] }
mongos> db.addUser('sa_read','123root_read',true) #添加uba管理员可读用户
WARNING: The 'addUser' shell helper is DEPRECATED. Please use 'createUser' instead
Successfully added user: { "user" : "sa_read", "roles" : [ "read" ] }
9、关闭三台服务器的mongod和mongos服务
killall mongos && killall mongod
10、成keyfile：(每个进程的key file都保持一致)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# openssl rand -base64 753 >keyfile
# chmod 600 keyfile #重点 必须
# cat keyfile
4a7w479ouV6fY5CASqxeWzzPhPLzZgYnHKWOFSIxjMABjaWK7up6Keb2+ZKz1C2R
tfl7ukCBelW2xOL0h3gOSAdIhUi0j8IMBYZV/kbUSSf0987xIur3lHZqUINPVVyl
ib+XPrg3SO4IG25tCQbEf/twvS1CxiDmi/GtQZfqOjUX7MEJK3ynFBsB/9wU1nf1
6rLhBPIdmSnpNf5s9yGohx2wn6dQcJ8o0wn75oOBysX2mPkLI0tEu4//7x1PgY9Q
Ws4XdXs1MFqUYXkh8K6RFd/TKB2zs5fonzqiXrgwpaW1KUFZB1lAfQwWePmxsWul
HqR96/yTGPxQKOVpm1Zdj2LGa616X3Www16Td4c2UVoM7mPi+Js8R2Flc6oV9H+t
eEDLMIVRDAAuQBNZl1reye0iaL8F0SgoELPiEm4A2La+ovNrK6K+shV+jmvrOEIC
6KJly68ZX6x+Xd5qxfoREYKsrwMJJYY1tWesUE+26QUs4TJB17uQzc1wJaRQK0pz
4B3E4DFzJT72ybaGFpbP4v9gMfoXXc6pPWEJ1LMvhvv4vNl5vDSpbS3YNUQjaTcU
juM2FlcUeKe3VFdqHkvdrhORv52gcrqnXco7UojOYmOHRpefXmL49XLlqw9FOJ2u
giGKSs06OaK7kyiVayUT4fUR+OXVOAAbPRgaNbDcI9P5knWkYwNdAwP+jH9h73nI
ZqdZ7SvDW0/iT6zbPfeQubSYJw//NG5gfN5Rsd5oEpXXz8bFnuJSC1ltzJenTYYN
0vsS/8EpUKhbLYjo5c93SBjopODBfA5pqQ9hRoL36l9SfWnUVgYaqpJgvKUmwb9P
1Bz1YCLRo1cU9skz+chhRnZWp2c2ElMfdMD/XpcAbgY1KToEJFAkTiNRHvgPYjiw
eGgEBSXlMBlKDkNqmLZ6i/C5fCWsqDEDrMPtBM4Pp9SOPjbkwpwYHxBCUPM7oyQT
fCisA72oOBB5L4vO55soTzX+jBBrOTBaJhdsEUMC0rEq
#将生成的keyfile 拷贝到mongod/mongos/config 进程对应的文件夹
# cp keyfile /data/mongodb/config/data/
# cp keyfile /data/mongodb/mongos/
# cp keyfile /data/mongodb/shard1/data/
# cp keyfile /data/mongodb/shard2/data/
# cp keyfile /data/mongodb/shard3/data/
然后通过scp的方式把这个文件分别拷贝到slave1和slave2机器的对应目录中
验证slave1服务器的keyfile文件
# find /data/mongodb -name keyfile
/data/mongodb/shard2/data/keyfile
/data/mongodb/shard1/data/keyfile
/data/mongodb/shard3/data/keyfile
/data/mongodb/config/data/keyfile
/data/mongodb/mongos/keyfile
验证slave2服务器的keyfile文件
#find /data/mongodb -name keyfile
/data/mongodb/shard2/data/keyfile
/data/mongodb/shard1/data/keyfile
/data/mongodb/shard3/data/keyfile
/data/mongodb/config/data/keyfile
/data/mongodb/mongos/keyfile
使用--keyFile参数指定前面生成的keyfile文件，重启三台机器全部mongod,mongos进程
1、分别在master、slave1和slave2服务器上启动配置服务器
mongod --configsvr --dbpath /data/mongodb/config/data --port 21000 --logpath /data/mongodb/config/log/config.log --fork --keyFile /data/mongodb/config/data/keyfile
#注意在这个过程中，会报错，只需要再次执行chmod 600 /data/mongodb/config/data/keyfile 就可以
2、分别在master、slave1和slave2服务器上启动mongos服务器
mongos--configdb 192.168.0.178:21000,192.168.0.180:21000,192.168.0.181:21000 --port 20000--keyFile /data/mongodb/mongos/keyfile --logpath/data/mongodb/mongos/log/mongos.log --fork
#执行chmod 600 /data/mongodb/mongos/keyfile
3、分别在master、slave1和slave2服务器上配置分片副本集
mongod --shardsvr --replSet shard1 --port 22001 --keyFile /data/mongodb/shard1/data/keyfile --dbpath /data/mongodb/shard1/data--logpath /data/mongodb/shard1/log/shard1.log --fork --nojournal--oplogSize 10
mongod --shardsvr --replSet shard2 --port 22002 --keyFile /data/mongodb/shard2/data/keyfile --dbpath /data/mongodb/shard2/data--logpath /data/mongodb/shard1/log/shard2.log --fork --nojournal--oplogSize 10
mongod --shardsvr --replSet shard3 --port 22003 --keyFile /data/mongodb/shard3/data/keyfile --dbpath /data/mongodb/shard3/data--logpath /data/mongodb/shard1/log/shard3.log --fork --nojournal--oplogSize 10
#执行以下
chmod 600 /data/mongodb/shard1/data/keyfile
chmod 600 /data/mongodb/shard2/data/keyfile
chmod 600 /data/mongodb/shard3/data/keyfile
验证认证
需要认证，没有认证没有任何权限
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# mongo localhost:20000
MongoDB shell version: 2.6.1
connecting to: localhost:20000/test
Error while trying to show server startup warnings: not authorized on admin to execute command { getLog: "startupWarnings" }
mongos> show collections;
2014-08-03T12:02:48.535+0800 error: {
"$err" : "not authorized for query on test.system.namespaces",
"code" : 13
} at src/mongo/shell/query.js:131
mongos> use uba
switched to db uba
mongos> db
uba
mongos> show collections;
2014-08-03T12:03:10.923+0800 error: { "$err" : "not authorized for query on uba.system.namespaces", "code" : 13 } at src/mongo/shell/query.js:131
账户认证后状态
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
#admin库认证
# mongo localhost:20000/admin -u admin -p allentuns
MongoDB shell version: 2.6.1
connecting to: localhost:20000/admin
mongos> show collections;
system.indexes
system.users
system.version
#uba库认证 读写权限用户
# mongo localhost:20000/uba -u sa -p 123root
MongoDB shell version: 2.6.1
connecting to: localhost:20000/uba
Error while trying to show server startup warnings: not authorized on admin to execute command { getLog: "startupWarnings" }
mongos> show collections;
system.indexes
table1
mongos> db.table1.count()
0
mongos> for (var i = 1; i <= 1000; i++)db.table1.save({id:i,name:"username"+i,age:i});
WriteResult({ "nInserted" : 1 })
mongos> db.table1.count()
1000
mongos> db.table1.findOne()
{
"_id" : ObjectId("53ddb52a65bb900f61e458a1"),
"id" : 1,
"name" : "username1",
"age" : 1
}
mongos> db.auth('sa','123root')
1
#uba库 只读权限用户
# mongo localhost:20000/uba -u sa_read -p 123root_read
MongoDB shell version: 2.6.1
connecting to: localhost:20000/uba
Error while trying to show server startup warnings: not authorized on admin to execute command { getLog: "startupWarnings" }
mongos> db.auth('sa_read','123root_read')
1
mongos> db
uba
mongos> for (var i = 1; i <= 1000; i++)db.table1.save({id:i,name:"username"+i,age:i});
WriteResult({
"writeError" : {
"code" : 13,
"errmsg" : "not authorized on uba to execute command { insert: \"table1\", documents: [ { id: 1000.0, name: \"username1000\", age: 1000.0, _id: ObjectId('53ddb5d62390d9a487087c87') } ], ordered: true }"
}
})
mongos> db.table1.findOne()
{
"_id" : ObjectId("53ddb52a65bb900f61e458a1"),
"id" : 1,
"name" : "username1",
"age" : 1
}
mongos> db.table1.count()
1000
验证主从切换，现在在192.168.0.180 slave1集群上停掉22001端口的服务，看master是否能接替slave1主的工作
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# netstat -tnlp |grep mongo
tcp 0 0 0.0.0.0:22001 0.0.0.0:* LISTEN 1955/mongod
tcp 0 0 0.0.0.0:22002 0.0.0.0:* LISTEN 2019/mongod
tcp 0 0 0.0.0.0:22003 0.0.0.0:* LISTEN 2081/mongod
tcp 0 0 0.0.0.0:20000 0.0.0.0:* LISTEN 1870/mongos
tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN 1847/mongod
# kill 1955
# netstat -tnlp |grep mongo
tcp 0 0 0.0.0.0:22002 0.0.0.0:* LISTEN 2019/mongod
tcp 0 0 0.0.0.0:22003 0.0.0.0:* LISTEN 2081/mongod
tcp 0 0 0.0.0.0:20000 0.0.0.0:* LISTEN 1870/mongos
tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN 1847/mongod
# mongo 192.168.0.178:22001/admin
MongoDB shell version: 2.6.1
connecting to: 192.168.0.178:22001/admin
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Error while trying to show server startup warnings: not authorized on admin to execute command { getLog: "startupWarnings" }
shard1:PRIMARY> ^C
bye
# mongo 192.168.0.178:22002/admin
MongoDB shell version: 2.6.1
connecting to: 192.168.0.178:22002/admin
Error while trying to show server startup warnings: not authorized on admin to execute command { getLog: "startupWarnings" }
shard2:ARBITER> ^C
bye
# mongod --shardsvr --replSet shard1 --port 22001 --keyFile /data/mongodb/shard1/data/keyfile --dbpath /data/mongodb/shard1/data--logpath /data/mongodb/shard1/log/shard1.log --fork --nojournal--oplogSize 10
about to fork child process, waiting until server is ready for connections.
forked process: 2641
child process started successfully, parent exiting
# mongo 192.168.0.180:22001/admin
MongoDB shell version: 2.6.1
connecting to: 192.168.0.180:22001/admin
Error while trying to show server startup warnings: not authorized on admin to execute command { getLog: "startupWarnings" }
shard1:SECONDARY>
页:
[1]