Linux与云计算——第二阶段 第五章：存储Storage服务器架设—分布式存储Ceph
Linux与云计算——第二阶段Linux服务器架设第五章：存储Storage服务器架设—分布式存储Ceph
http://s1.运维网.com/wyfs02/M02/85/48/wKiom1efDNqCFrz7AAvpWirnkoc650.jpg-wh_500x0-wm_3-wmp_4-s_1944763573.jpg
1 Ceph 配置Ceph集群
Install Distributed File System "Ceph" to Configure Storage Cluster.
For example on here, Configure Cluster with 1 admin Node and 3 Storage Node like follows.
|
+--------------------+ | +-------------------+
| |10.0.0.30| 10.0.0.x| [ Client] |
| Ceph-Deploy +-----------+-----------+ |
| | | | |
+--------------------+ | +-------------------+
+----------------------------+----------------------------+
| | |
|10.0.0.51 |10.0.0.52 |10.0.0.53
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| | | | | |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| | | | | |
+-----------------------+ +-----------------------+ +-----------------------+
Add a user for Ceph admin on all Nodes.
It adds "cent" user on this exmaple.
Grant root priviledge to Ceph admin user just added above with sudo settings.
And also install required packages.
Furthermore, If Firewalld is running on all Nodes, allow SSH service.
Set all of above on all Nodes.
# echo -e 'Defaults:cent !requiretty\ncent ALL = (root) NOPASSWD:ALL' | tee /etc/sudoers.d/ceph
# chmod 440 /etc/sudoers.d/ceph
# yum -y install centos-release-ceph-hammer epel-release yum-plugin-priorities
# sed -i -e "s/enabled=1/enabled=1\npriority=1/g" /etc/yum.repos.d/CentOS-Ceph-Hammer.repo
# firewall-cmd --add-service=ssh --permanent
# firewall-cmd –reload
On Monitor Node (Monitor Daemon), If Firewalld is running, allow required port.
# firewall-cmd --add-port=6789/tcp --permanent
# firewall-cmd –reload
On Storage Nodes (Object Storage), If Firewalld is running, allow required ports.
# firewall-cmd --add-port=6800-7100/tcp --permanent
# firewall-cmd –reload
Login as a Ceph admin user and configure Ceph.
Set SSH key-pair from Ceph Admin Node (it's "dlp.srv.world" on this example) to all storage Nodes.
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/cent/.ssh/id_rsa):
Created directory '/home/cent/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/cent/.ssh/id_rsa.
Your public key has been saved in /home/cent/.ssh/id_rsa.pub.
The key fingerprint is:
54:c3:12:0e:d3:65:11:49:11:73:35:1b:e3:e8:63:5a cent@dlp.srv.world
The key's randomart image is:
$ vi ~/.ssh/config
# create new ( define all nodes and users )
Host dlp
Hostname dlp.srv.world
User cent
Host node01
Hostname node01.srv.world
User cent
Host node02
Hostname node02.srv.world
User cent
Host node03
Hostname node03.srv.world
User cent
$ chmod 600 ~/.ssh/config
# transfer key file
$ ssh-copy-id node01
cent@node01.srv.world's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'node01'"
and check to make sure that only the key(s) you wanted were added.
$ ssh-copy-id node02
$ ssh-copy-id node03
Install Ceph to all Nodes from Admin Node.
$ sudo yum -y install ceph-deploy
$ mkdir ceph
$ cd ceph
$ ceph-deploy new node01
$ vi ./ceph.conf
# add to the end
osd pool default size = 2
# Install Ceph on each Node
$ ceph-deploy install dlp node01 node02 node03
# settings for monitoring and keys
$ ceph-deploy mon create-initial
Configure Ceph Cluster from Admin Node.
Beforeit, Create a directory /storage01 on Node01, /storage02 on Node02, /storage03 on node03 on this example.
# prepare Object Storage Daemon
$ ceph-deploy osd prepare node01:/storage01 node02:/storage02 node03:/storage03
# activate Object Storage Daemon
$ ceph-deploy osd activate node01:/storage01 node02:/storage02 node03:/storage03
# transfer config files
$ ceph-deploy admin dlp node01 node02 node03
$ sudo chmod 644 /etc/ceph/ceph.client.admin.keyring
# show status (display like follows if no ploblem)
$ ceph health
HEALTH_OK
By the way, if you'd like to clean settings and re-configure again, do like follows.
# remove packages
$ ceph-deploy purge dlp node01 node02 node03
# remove settings
$ ceph-deploy purgedata dlp node01 node02 node03
$ ceph-deploy forgetkeys
2 作为Block设备使用
Configure Clients to use Ceph Storage like follows.
|
+--------------------+ | +-------------------+
| |10.0.0.30| 10.0.0.x| [ Client] |
| Ceph-Deploy +-----------+-----------+ |
| | | | |
+--------------------+ | +-------------------+
+----------------------------+----------------------------+
| | |
|10.0.0.51 |10.0.0.52 |10.0.0.53
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| | | | | |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| | | | | |
+-----------------------+ +-----------------------+ +-----------------------+
For exmaple, Create a block device and mount it on a Client.
First, Configure Sudo and SSH key-pair for a user on a Client and next, Install Ceph from Ceph admin Node like follows.
$ ceph-deploy install client
$ ceph-deploy admin client
Create a Block device and mount it on a Client.
$ sudo chmod 644 /etc/ceph/ceph.client.admin.keyring
# create a disk with 10G
$ rbd create disk01 --size 10240
# show list
$ rbd ls -l
NAME SIZE PARENT FMT PROT LOCK
disk01 10240M 2
# map the image to device
$ sudo rbd map disk01
/dev/rbd0
# show mapping
$ rbd showmapped
id pool imagesnap device
0rbddisk01 - /dev/rbd0
# format with XFS
$ sudo mkfs.xfs /dev/rbd0
# mount device
$ sudo mount /dev/rbd0 /mnt
$ df -hT
Filesystem Type SizeUsed Avail Use% Mounted on
/dev/mapper/centos-root xfs 27G1.3G 26G 5% /
devtmpfs devtmpfs2.0G 02.0G 0% /dev
tmpfs tmpfs 2.0G 02.0G 0% /dev/shm
tmpfs tmpfs 2.0G8.4M2.0G 1% /run
tmpfs tmpfs 2.0G 02.0G 0% /sys/fs/cgroup
/dev/vda1 xfs 497M151M347M31% /boot
/dev/rbd0 xfs 10G 33M 10G 1% /mnt
3 作为文件系统使用
Configure Clients to use Ceph Storage like follows.
|
+--------------------+ | +-------------------+
| |10.0.0.30| 10.0.0.x| [ Client] |
| Ceph-Deploy +-----------+-----------+ |
| | | | |
+--------------------+ | +-------------------+
+----------------------------+----------------------------+
| | |
|10.0.0.51 |10.0.0.52 |10.0.0.53
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| | | | | |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| | | | | |
+-----------------------+ +-----------------------+ +-----------------------+
For example, mount as Filesystem on a Client.
Create MDS (MetaData Server) on a Node which you'd like to set MDS. It sets to node01 on this exmaple.
$ ceph-deploy mds create node01
Create at least 2 RADOS pools on MDS Node and activate MetaData Server.
For pg_num which is specified at the end of a creating command, refer to official document and decide appropriate value.
http://docs.ceph.com/docs/master/rados/operations/placement-groups/
$ sudo chmod 644 /etc/ceph/ceph.client.admin.keyring
# create pools
$ ceph osd pool create cephfs_data 128
pool 'cephfs_data' created
$ ceph osd pool create cephfs_metadata 128
pool 'cephfs_metadata' created
# enable pools
$ ceph fs new cephfs cephfs_metadata cephfs_data
new fs with metadata pool 2 and data pool 1
# show list
$ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools:
$ ceph mds stat
e5: 1/1/1 up {0=node01=up:active}
Mount CephFS on a Client.
# yum -y install ceph-fuse
# get admin key
# ssh cent@node01.srv.world "sudo ceph-authtool -p /etc/ceph/ceph.client.admin.keyring" > admin.key
cent@node01.srv.world's password:
# chmod 600 admin.key
# mount -t ceph node01.srv.world:6789:/ /mnt -o name=admin,secretfile=admin.key
# df -hT
Filesystem Type SizeUsed Avail Use% Mounted on
/dev/mapper/centos-root xfs 27G1.3G 26G 5% /
devtmpfs devtmpfs2.0G 02.0G 0% /dev
tmpfs tmpfs 2.0G 02.0G 0% /dev/shm
tmpfs tmpfs 2.0G8.3M2.0G 1% /run
tmpfs tmpfs 2.0G 02.0G 0% /sys/fs/cgroup
/dev/vda1 xfs 497M151M347M31% /boot
10.0.0.51:6789:/ ceph 80G 19G 61G24% /mnt
详细视频课程请戳—→ http://edu.运维网.com/course/course_id-6574.html
页:
[1]