ceph详细安装部署教程（多监控节点）
一、前期准备安装ceph-deploy工具所有的服务器都是用root用户登录的
1、安装环境
系统centos-6.5
设备：1台admin-node (ceph-ploy)1台 monistor 2台 osd
2、关闭所有节点的防火墙及关闭selinux，重启机器。
service iptables stop
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
chkconfig iptables off
3、编辑admin-node节点的ceph yum仓库
vi /etc/yum.repos.d/ceph.repo
name=Ceph noarch packages
baseurl=http://ceph.com/rpm/el6/noarch/
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
4、安装搜狐的epel仓库
rpm -ivh http://mirrors.sohu.com/fedora-epel/6/x86_64/epel-release-6-8.noarch.rpm
5、更新admin-node节点的yum源
yum clean all
yum update -y
6、在admin-node节点上建立一个ceph集群目录
mkdir /ceph
cd/ceph
7、在admin-node节点上安装ceph部署工具
yum install ceph-deploy -y
8、配置admin-node节点的hosts文件
vi /etc/hosts
10.240.240.210 admin-node
10.240.240.211 node1
10.240.240.212 node2
10.240.240.213 node3
二、配置ceph-deploy部署的无密码登录每个ceph节点
1、在每个Ceph节点上安装一个SSH服务器
$ yum install openssh-server -y
2、配置您的admin-node管理节点与每个Ceph节点无密码的SSH访问。
# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
3、复制admin-node节点的秘钥到每个ceph节点
ssh-copy-id root@admin-node
ssh-copy-id root@node1
ssh-copy-id root@node2
ssh-copy-id root@node3
4、测试每台ceph节点不用密码是否可以登录
ssh root@node1
ssh root@node2
ssh root@node3
5、修改admin-node管理节点的~/.ssh / config文件，这样它登录到Ceph节点创建的用户
Host admin-node
Hostname admin-node
User root
Host node1
Hostname node1
User root
Host node2
Hostname node2
User root
Host node3
Hostname node3
User root
三、用ceph-deploy工具部署ceph集群
1、在admin-node节点上新建一个ceph集群
#ceph-deploy new node1 node2 node3 （执行这条命令后node1 node2 node3都作为了monitor节点，多个mon节点可以实现互备）
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy new node1 node2 node3
Creating new cluster named ceph
Resolving host node1
Monitor node1 at 10.240.240.211
making sure passwordless SSH succeeds
connected to host: admin-node
Running command: ssh -CT -o BatchMode=yes node1
Resolving host node2
Monitor node2 at 10.240.240.212
making sure passwordless SSH succeeds
connected to host: admin-node
Running command: ssh -CT -o BatchMode=yes node2
Resolving host node3
Monitor node3 at 10.240.240.213
making sure passwordless SSH succeeds
connected to host: admin-node
Running command: ssh -CT -o BatchMode=yes node3
Monitor initial members are ['node1', 'node2', 'node3']
Monitor addrs are ['10.240.240.211', '10.240.240.212', '10.240.240.213']
Creating a random mon key...
Writing initial config to ceph.conf...
Writing monitor keyring to ceph.mon.keyring...
查看生成的文件
# ls
ceph.confceph.logceph.mon.keyring
查看ceph的配置文件，三个节点都变为了控制节点
# cat ceph.conf
auth_service_required = cephx
filestore_xattr_use_omap = true
auth_client_required = cephx
auth_cluster_required = cephx
mon_host = 10.240.240.211,10.240.240.212,10.240.240.213
mon_initial_members = node1, node2, node3
fsid = 4dc38af6-f628-4c1f-b708-9178cf4e032b
#
2、部署之前确保ceph每个节点没有ceph数据包（先清空之前所有的ceph数据，如果是新装不用执行此步骤，如果是重新部署的话也执行下面的命令）
# ceph-deploy purgedata admin-node node1 node2 node3
# ceph-deploy forgetkeys
# ceph-deploy purge admin-node node1 node2 node3
如果是新装的话是没有任何数据的
3、编辑admin-node节点的ceph配置文件，把下面的配置放入ceph.conf中
osd pool default size = 2
4、在admin-node节点用ceph-deploy工具向各个节点安装ceph
# ceph-deploy install admin-node node1 node2 node3
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy install admin-node node1 node2 node3
Installing stable version firefly on cluster ceph hosts admin-node node1 node2 node3
Detecting platform for host admin-node ...
connected to host: admin-node
detect platform information from remote host
detect machine type
Distro info: CentOS 6.5 Final
installing ceph on admin-node
Running command: yum clean all
Loaded plugins: fastestmirror, refresh-packagekit, security
Cleaning repos: Ceph Ceph-noarch base ceph-source epel extras updates
Cleaning up Everything
Cleaning up list of fastest mirrors
Running command: yum -y install wget
Loaded plugins: fastestmirror, refresh-packagekit, security
Determining fastest mirrors
* base: mirrors.btte.net
* epel: mirrors.neusoft.edu.cn
* extras: mirrors.btte.net
* updates: mirrors.btte.net
Setting up Install Process
Package wget-1.12-1.11.el6_5.x86_64 already installed and latest version
Nothing to do
adding EPEL repository
Running command: wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
--2014-06-07 22:05:34--http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Resolving dl.fedoraproject.org... 209.132.181.24, 209.132.181.25, 209.132.181.26, ...
Connecting to dl.fedoraproject.org|209.132.181.24|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14540 (14K)
Saving to: `epel-release-6-8.noarch.rpm.1'
0K .......... .... 100% 73.8K=0.2s
2014-06-07 22:05:35 (73.8 KB/s) - `epel-release-6-8.noarch.rpm.1' saved
Running command: rpm -Uvh --replacepkgs epel-release-6*.rpm
Preparing... ##################################################
epel-release ##################################################
Running command: rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
Running command: rpm -Uvh --replacepkgs http://ceph.com/rpm-firefly/el6/noarch/ceph-release-1-0.el6.noarch.rpm
Retrieving http://ceph.com/rpm-firefly/el6/noarch/ceph-release-1-0.el6.noarch.rpm
Preparing... ##################################################
ceph-release ##################################################
Running command: yum -y -q install ceph
Package ceph-0.80.1-2.el6.x86_64 already installed and latest version
Running command: ceph --version
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
Detecting platform for host node1 ...
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
installing ceph on node1
Running command: yum clean all
Loaded plugins: fastestmirror, refresh-packagekit, security
Cleaning repos: base extras updates
Cleaning up Everything
Cleaning up list of fastest mirrors
Running command: yum -y install wget
Loaded plugins: fastestmirror, refresh-packagekit, security
Determining fastest mirrors
* base: mirrors.btte.net
* extras: mirrors.btte.net
* updates: mirrors.btte.net
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 0:1.12-1.8.el6 will be updated
---> Package wget.x86_64 0:1.12-1.11.el6_5 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Updating:
wget x86_64 1.12-1.11.el6_5 updates 483 k
Transaction Summary
================================================================================
Upgrade 1 Package(s)
Total download size: 483 k
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : wget-1.12-1.11.el6_5.x86_64 1/2
Cleanup : wget-1.12-1.8.el6.x86_64 2/2
Verifying: wget-1.12-1.11.el6_5.x86_64 1/2
Verifying: wget-1.12-1.8.el6.x86_64 2/2
Updated:
wget.x86_64 0:1.12-1.11.el6_5
Complete!
adding EPEL repository
Running command: wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
--2014-06-07 22:06:57--http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Resolving dl.fedoraproject.org... 209.132.181.23, 209.132.181.24, 209.132.181.25, ...
Connecting to dl.fedoraproject.org|209.132.181.23|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14540 (14K)
Saving to: `epel-release-6-8.noarch.rpm'
0K .......... .... 100% 69.6K=0.2s
2014-06-07 22:06:58 (69.6 KB/s) - `epel-release-6-8.noarch.rpm' saved
Running command: rpm -Uvh --replacepkgs epel-release-6*.rpm
Preparing... ##################################################
epel-release ##################################################
warning: epel-release-6-8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Running command: rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
Running command: rpm -Uvh --replacepkgs http://ceph.com/rpm-firefly/el6/noarch/ceph-release-1-0.el6.noarch.rpm
Retrieving http://ceph.com/rpm-firefly/el6/noarch/ceph-release-1-0.el6.noarch.rpm
Preparing... ##################################################
ceph-release ##################################################
Running command: yum -y -q install ceph
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Importing GPG key 0x0608B895:
Userid : EPEL (6)
Package: epel-release-6-8.noarch (installed)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Warning: RPMDB altered outside of yum.
Running command: ceph --version
Traceback (most recent call last):
File "/usr/bin/ceph", line 53, in
import argparse
ImportError: No module named argparse
RuntimeError: command returned non-zero exit status: 1
RuntimeError: Failed to execute command: ceph --version
上面报错信息的解决方法是：在报错的节点上执行下面的命令
# yum install *argparse* -y
5、添加初始监控节点并收集密钥(新的ceph-deploy v1.1.3以后的版本)。
# ceph-deploy mon create-initial
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy mon create-initial
Deploying mon, cluster ceph hosts node1
detecting platform for host node1 ...
connected to host: node1
detect platform information from remote host
detect machine type
distro info: CentOS 6.4 Final
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to node1
get remote short hostname
remote hostname: node1
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-node1/done
done path does not exist: /var/lib/ceph/mon/ceph-node1/done
creating keyring file: /var/lib/ceph/tmp/ceph-node1.mon.keyring
create the monitor keyring file
Running command: ceph-mon --cluster ceph --mkfs -i node1 --keyring /var/lib/ceph/tmp/ceph-node1.mon.keyring
ceph-mon: mon.noname-a 10.240.240.211:6789/0 is local, renaming to mon.node1
ceph-mon: set fsid to 369daf5a-e844-4e09-a9b1-46bb985aec79
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-node1 for mon.node1
unlinking keyring file /var/lib/ceph/tmp/ceph-node1.mon.keyring
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
locating the `service` executable...
Running command: /sbin/service ceph -c /etc/ceph/ceph.conf start mon.node1
/etc/init.d/ceph: line 15: /lib/lsb/init-functions: No such file or directory
RuntimeError: command returned non-zero exit status: 1
Failed to execute command: /sbin/service ceph -c /etc/ceph/ceph.conf start mon.node1
GenericError: Failed to create 1 monitors
解决上面报错信息的方法：
手动在node1 node2 node3节点上执行下面的命令
# yum install redhat-lsb-y
再次执行上面的命令可以成功激活监控节点
# ceph-deploy mon create-initial
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy mon create-initial
Deploying mon, cluster ceph hosts node1 node2 node3
detecting platform for host node1 ...
connected to host: node1
detect platform information from remote host
detect machine type
distro info: CentOS 6.4 Final
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to node1
get remote short hostname
remote hostname: node1
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-node1/done
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
locating the `service` executable...
Running command: /sbin/service ceph -c /etc/ceph/ceph.conf start mon.node1
=== mon.node1 ===
Starting Ceph mon.node1 on node1...already running
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
********************************************************************************
status for monitor: mon.node1
{
"election_epoch": 6,
"extra_probe_peers": [
"10.240.240.212:6789/0",
"10.240.240.213:6789/0"
],
"monmap": {
"created": "0.000000",
"epoch": 2,
"fsid": "4dc38af6-f628-4c1f-b708-9178cf4e032b",
"modified": "2014-06-07 22:38:29.435203",
"mons": [
{
"addr": "10.240.240.211:6789/0",
"name": "node1",
"rank": 0
},
{
"addr": "10.240.240.212:6789/0",
"name": "node2",
"rank": 1
},
{
"addr": "10.240.240.213:6789/0",
"name": "node3",
"rank": 2
}
]
},
"name": "node1",
"outside_quorum": [],
"quorum": [
0,
1,
2
],
"rank": 0,
"state": "leader",
"sync_provider": []
}
********************************************************************************
monitor: mon.node1 is running
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
detecting platform for host node2 ...
connected to host: node2
detect platform information from remote host
detect machine type
distro info: CentOS 6.4 Final
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to node2
get remote short hostname
remote hostname: node2
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-node2/done
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
locating the `service` executable...
Running command: /sbin/service ceph -c /etc/ceph/ceph.conf start mon.node2
=== mon.node2 ===
Starting Ceph mon.node2 on node2...already running
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status
********************************************************************************
status for monitor: mon.node2
{
"election_epoch": 6,
"extra_probe_peers": [
"10.240.240.211:6789/0",
"10.240.240.213:6789/0"
],
"monmap": {
"created": "0.000000",
"epoch": 2,
"fsid": "4dc38af6-f628-4c1f-b708-9178cf4e032b",
"modified": "2014-06-07 22:38:29.435203",
"mons": [
{
"addr": "10.240.240.211:6789/0",
"name": "node1",
"rank": 0
},
{
"addr": "10.240.240.212:6789/0",
"name": "node2",
"rank": 1
},
{
"addr": "10.240.240.213:6789/0",
"name": "node3",
"rank": 2
}
]
},
"name": "node2",
"outside_quorum": [],
"quorum": [
0,
1,
2
],
"rank": 1,
"state": "peon",
"sync_provider": []
}
********************************************************************************
monitor: mon.node2 is running
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status
detecting platform for host node3 ...
connected to host: node3
detect platform information from remote host
detect machine type
distro info: CentOS 6.4 Final
determining if provided host has same hostname in remote
get remote short hostname
deploying mon to node3
get remote short hostname
remote hostname: node3
write cluster configuration to /etc/ceph/{cluster}.conf
create the mon path if it does not exist
checking for done path: /var/lib/ceph/mon/ceph-node3/done
create a done file to avoid re-doing the mon deployment
create the init path if it does not exist
locating the `service` executable...
Running command: /sbin/service ceph -c /etc/ceph/ceph.conf start mon.node3
=== mon.node3 ===
Starting Ceph mon.node3 on node3...already running
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node3.asok mon_status
********************************************************************************
status for monitor: mon.node3
{
"election_epoch": 6,
"extra_probe_peers": [
"10.240.240.211:6789/0",
"10.240.240.212:6789/0"
],
"monmap": {
"created": "0.000000",
"epoch": 2,
"fsid": "4dc38af6-f628-4c1f-b708-9178cf4e032b",
"modified": "2014-06-07 22:38:29.435203",
"mons": [
{
"addr": "10.240.240.211:6789/0",
"name": "node1",
"rank": 0
},
{
"addr": "10.240.240.212:6789/0",
"name": "node2",
"rank": 1
},
{
"addr": "10.240.240.213:6789/0",
"name": "node3",
"rank": 2
}
]
},
"name": "node3",
"outside_quorum": [],
"quorum": [
0,
1,
2
],
"rank": 2,
"state": "peon",
"sync_provider": []
}
********************************************************************************
monitor: mon.node3 is running
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node3.asok mon_status
processing monitor mon.node1
connected to host: node1
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node1.asok mon_status
mon.node1 monitor has reached quorum!
processing monitor mon.node2
connected to host: node2
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node2.asok mon_status
mon.node2 monitor has reached quorum!
processing monitor mon.node3
connected to host: node3
Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node3.asok mon_status
mon.node3 monitor has reached quorum!
all initial monitors are running and have formed quorum
Running gatherkeys...
Checking node1 for /etc/ceph/ceph.client.admin.keyring
connected to host: node1
detect platform information from remote host
detect machine type
fetch remote file
Got ceph.client.admin.keyring key from node1.
Have ceph.mon.keyring
Checking node1 for /var/lib/ceph/bootstrap-osd/ceph.keyring
connected to host: node1
detect platform information from remote host
detect machine type
fetch remote file
Got ceph.bootstrap-osd.keyring key from node1.
Checking node1 for /var/lib/ceph/bootstrap-mds/ceph.keyring
connected to host: node1
detect platform information from remote host
detect machine type
fetch remote file
Got ceph.bootstrap-mds.keyring key from node1.
通过上面的输出信息可知，三个节点都变为了监控节点
查看ceph集群目录多了下面几个文件
ceph.bootstrap-mds.keyring
ceph.bootstrap-osd.keyring
ceph.client.admin.keyring
6、添加osd节点
先添加node1节点，进入node1节点查看未分配的分区
# ssh node1
# fdisk -l
Disk /dev/sda: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d6653
Device Boot Start End Blocks IdSystem
/dev/sda1 * 1 39 307200 83Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 39 6401 51104768 83Linux
/dev/sda3 6401 6528 1015808 82Linux swap / Solaris
Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x843e46d0
Device Boot Start End Blocks IdSystem
/dev/sdb1 1 2610 20964793+ 5Extended
/dev/sdb5 1 2610 20964762 83Linux
# df -h
Filesystem SizeUsed Avail Use% Mounted on
/dev/sda2 48G2.5G 44G 6% /
tmpfs 242M 68K242M 1% /dev/shm
/dev/sda1 291M 33M243M12% /boot
查看可以看出第二块硬盘为使用，使用第二块硬盘的sdb5分区作为osd硬盘
在admin-node节点上添加osd设备
#ceph-deploy osd prepare node1:/dev/sdb5
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy osd prepare node2:/var/local/osd0 node3:/var/local/osd1
Preparing cluster ceph disks node2:/var/local/osd0: node3:/var/local/osd1:
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.5 Final
Deploying osd to node1
write cluster configuration to /etc/ceph/{cluster}.conf
osd keyring does not exist yet, creating one
create a keyring file
IOError: No such file or directory: '/var/lib/ceph/bootstrap-osd/ceph.keyring'
GenericError: Failed to create 1 OSDs
解决上面报错信息的方法如下：（上面的报错一般出现在添加非监控节点的osd）
上面错误信息的意思是：在创建osd节点的时候在osd节点上缺少/var/lib/ceph/bootstrap-osd/ceph.keyring文件，查看监控节点发现有这个文件，把监控节点上的文件拷贝到node1节点上去即可。
在node1节点上建立一个目录：mkdir /var/lib/ceph/bootstrap-osd/。
登录node1：
# ssh node1
#scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@node1:/var/lib/ceph/bootstrap-osd/
再次执行osd初始化命令
# ceph-deploy osd prepare node1:/dev/sdb5
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy osd prepare node1:/dev/sdb5
Preparing cluster ceph disks node1:/dev/sdb5:
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
Deploying osd to node1
write cluster configuration to /etc/ceph/{cluster}.conf
Running command: udevadm trigger --subsystem-match=block --action=add
Preparing host node1 disk /dev/sdb5 journal None activate False
Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdb5
mkfs.xfs: No such file or directory
ceph-disk: Error: Command '['/sbin/mkfs', '-t', 'xfs', '-f', '-i', 'size=2048', '--', '/dev/sdb5']' returned non-zero exit status 1
RuntimeError: command returned non-zero exit status: 1
Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdb5
GenericError: Failed to create 1 OSDs
上面的报错信息说明在node1上没有mkfs.xfs文件，需要在node1上安装mkfs.xfs文件。
# ssh node1
# yum install xfs* -y
再次执行osd初始化命令可以成功初始化新加入的osd节点
# ceph-deploy osd prepare node1:/dev/sdb5
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy osd prepare node1:/dev/sdb5
Preparing cluster ceph disks node1:/dev/sdb5:
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
Deploying osd to node1
write cluster configuration to /etc/ceph/{cluster}.conf
Running command: udevadm trigger --subsystem-match=block --action=add
Preparing host node1 disk /dev/sdb5 journal None activate False
Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdb5
meta-data=/dev/sdb5 isize=2048 agcount=4, agsize=1310298 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=5241190, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
INFO:ceph-disk:calling partx on prepared device /dev/sdb5
INFO:ceph-disk:re-reading known partitions will display errors
last arg is not the whole disk
call: partx -opts device wholedisk
checking OSD status...
Running command: ceph --cluster=ceph osd stat --format=json
Host node1 is now ready for osd use.
Unhandled exception in thread started by
Error in sys.excepthook:
在admin节点上激活osd设备
# ceph-deploy osd activate node1:/dev/sdb5
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy osd activate node1:/dev/sdb5
Activating cluster ceph disks node1:/dev/sdb5:
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
activating host node1 disk /dev/sdb5
will use init type: sysvinit
Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb5
got monmap epoch 2
2014-06-07 23:36:52.377131 7f1b9a7087a0 -1 journal FileJournal::_open: disabling aio for non-block journal.Use journal_force_aio to force use of aio anyway
2014-06-07 23:36:52.436136 7f1b9a7087a0 -1 journal FileJournal::_open: disabling aio for non-block journal.Use journal_force_aio to force use of aio anyway
2014-06-07 23:36:52.437990 7f1b9a7087a0 -1 filestore(/var/lib/ceph/tmp/mnt.LvzAgX) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-06-07 23:36:52.470988 7f1b9a7087a0 -1 created object store /var/lib/ceph/tmp/mnt.LvzAgX journal /var/lib/ceph/tmp/mnt.LvzAgX/journal for osd.0 fsid 4dc38af6-f628-4c1f-b708-9178cf4e032b
2014-06-07 23:36:52.471176 7f1b9a7087a0 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.LvzAgX/keyring: can't open /var/lib/ceph/tmp/mnt.LvzAgX/keyring: (2) No such file or directory
2014-06-07 23:36:52.471528 7f1b9a7087a0 -1 created new key in keyring /var/lib/ceph/tmp/mnt.LvzAgX/keyring
added key for osd.0
ERROR:ceph-disk:Failed to activate
Traceback (most recent call last):
File "/usr/sbin/ceph-disk", line 2579, in
main()
File "/usr/sbin/ceph-disk", line 2557, in main
args.func(args)
File "/usr/sbin/ceph-disk", line 1910, in main_activate
init=args.mark_init,
File "/usr/sbin/ceph-disk", line 1724, in mount_activate
mount_options=mount_options,
File "/usr/sbin/ceph-disk", line 1544, in move_mount
maybe_mkdir(osd_data)
File "/usr/sbin/ceph-disk", line 220, in maybe_mkdir
os.mkdir(*a, **kw)
OSError: No such file or directory: '/var/lib/ceph/osd/ceph-0'
RuntimeError: command returned non-zero exit status: 1
RuntimeError: Failed to execute command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb5
上面报错信息的意思是：在node1节点上没有/var/lib/ceph/osd/ceph-0这个目录，需要在node1节点上创建这个目录。
# ssh node1
# mkdir /var/lib/ceph/osd/
# mkdir /var/lib/ceph/osd/ceph-0
再次执行激活osd命令
# ceph-deploy osd activate node1:/dev/sdb5
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy osd activate node1:/dev/sdb5
Activating cluster ceph disks node1:/dev/sdb5:
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
activating host node1 disk /dev/sdb5
will use init type: sysvinit
Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb5
/etc/init.d/ceph: line 15: /lib/lsb/init-functions: No such file or directory
ceph-disk: Error: ceph osd start failed: Command '['/sbin/service', 'ceph', 'start', 'osd.0']' returned non-zero exit status 1
RuntimeError: command returned non-zero exit status: 1
RuntimeError: Failed to execute command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb5
上面报错信息的解决方法：
# ssh node1
yum install redhat-lsb-y
再次执行激活osd命令osd节点可以正常运行
# ceph-deploy osd activate node1:/dev/sdb5
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy osd activate node1:/dev/sdb5
Activating cluster ceph disks node1:/dev/sdb5:
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
activating host node1 disk /dev/sdb5
will use init type: sysvinit
Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb5
=== osd.0 ===
Starting Ceph osd.0 on node1...
starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
create-or-move updating item name 'osd.0' weight 0.02 at location {host=node1,root=default} to crush map
checking OSD status...
Running command: ceph --cluster=ceph osd stat --format=json
Unhandled exception in thread started by
Error in sys.excepthook:
Original exception was:
按上面的方法添加node2 node3为osd节点
7、复制ceph配置文件及密钥到mon、osd节点
# ceph-deploy admin admin-node node1 node2 node3
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy admin admin-node node1 node2 node3
Pushing admin keys and conf to admin-node
connected to host: admin-node
detect platform information from remote host
detect machine type
get remote short hostname
write cluster configuration to /etc/ceph/{cluster}.conf
Pushing admin keys and conf to node1
connected to host: node1
detect platform information from remote host
detect machine type
get remote short hostname
write cluster configuration to /etc/ceph/{cluster}.conf
Pushing admin keys and conf to node2
connected to host: node2
detect platform information from remote host
detect machine type
get remote short hostname
write cluster configuration to /etc/ceph/{cluster}.conf
Pushing admin keys and conf to node3
connected to host: node3
detect platform information from remote host
detect machine type
get remote short hostname
write cluster configuration to /etc/ceph/{cluster}.conf
Unhandled exception in thread started by
Error in sys.excepthook:
Original exception was:
8、确保你有正确的ceph.client.admin.keyring权限
# chmod +r /etc/ceph/ceph.client.admin.keyring
9、查看三台监控节点的选举状态
# ceph quorum_status --format json-pretty
{ "election_epoch": 30,
"quorum": [
0,
1,
2],
"quorum_names": [
"node1",
"node2",
"node3"],
"quorum_leader_name": "node1",
"monmap": { "epoch": 2,
"fsid": "4dc38af6-f628-4c1f-b708-9178cf4e032b",
"modified": "2014-06-07 22:38:29.435203",
"created": "0.000000",
"mons": [
{ "rank": 0,
"name": "node1",
"addr": "10.240.240.211:6789\/0"},
{ "rank": 1,
"name": "node2",
"addr": "10.240.240.212:6789\/0"},
{ "rank": 2,
"name": "node3",
"addr": "10.240.240.213:6789\/0"}]}}
10、查看集群运行状态
# ceph health
HEALTH_WARN clock skew detected on mon.node2, mon.node3
出现上面信息的意思是，node1 node2 node3的时间不一致，必须把他们的时间同步，解决方法如下：
把admin-node配置ntp服务器，所有的节点都同步admin-node。
再次执行结果如下：
# ceph health
HEALTH_OK
12、添加一个元数据服务器
# ceph-deploy mds create node1
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy mds create node1
Deploying mds, cluster ceph hosts node1:node1
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
remote host will use sysvinit
deploying mds bootstrap to node1
write cluster configuration to /etc/ceph/{cluster}.conf
create path if it doesn't exist
OSError: No such file or directory: '/var/lib/ceph/mds/ceph-node1'
GenericError: Failed to create 1 MDSs
解决上面报错的方法：
# ssh node1
Last login: Fri Jun6 06:41:25 2014 from 10.241.10.2
# mkdir /var/lib/ceph/mds/
# mkdir /var/lib/ceph/mds/ceph-node1
再次执行元数据服务器创建完成
# ceph-deploy mds create node1
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy mds create node1
Deploying mds, cluster ceph hosts node1:node1
connected to host: node1
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
remote host will use sysvinit
deploying mds bootstrap to node1
write cluster configuration to /etc/ceph/{cluster}.conf
create path if it doesn't exist
Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node1 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node1/keyring
Running command: service ceph start mds.node1
=== mds.node1 ===
Starting Ceph mds.node1 on node1...
starting mds.node1 at :/0
再次查看运行状态
# ceph -w
cluster 591ef1f4-69f7-442f-ba7b-49cdf6695656
health HEALTH_OK
monmap e1: 1 mons at {node1=10.240.240.211:6789/0}, election epoch 2, quorum 0 node1
mdsmap e4: 1/1/1 up {0=node1=up:active}
osdmap e9: 2 osds: 2 up, 2 in
pgmap v22: 192 pgs, 3 pools, 1884 bytes data, 20 objects
10310 MB used, 30616 MB / 40926 MB avail
192 active+clean
2014-06-06 08:12:49.021472 mon.0 pgmap v22: 192 pgs: 192 active+clean; 1884 bytes data, 10310 MB used, 30616 MB / 40926 MB avail; 10 B/s wr, 0 op/s
2014-06-06 08:14:47.932311 mon.0 pgmap v23: 192 pgs: 192 active+clean; 1884 bytes data, 10310 MB used, 30615 MB / 40926 MB avail
13、安装ceph client
安装ceph客户端
# ceph-deploy install ceph-client
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy install ceph-client
Installing stable version firefly on cluster ceph hosts ceph-client
Detecting platform for host ceph-client ...
connected to host: ceph-client
detect platform information from remote host
detect machine type
Distro info: CentOS 6.4 Final
installing ceph on ceph-client
Running command: yum clean all
Loaded plugins: fastestmirror, refresh-packagekit, security
Cleaning repos: Ceph Ceph-noarch base ceph-source epel extras updates
Cleaning up Everything
Cleaning up list of fastest mirrors
Running command: yum -y install wget
Loaded plugins: fastestmirror, refresh-packagekit, security
Determining fastest mirrors
* base: mirrors.btte.net
* epel: mirrors.hust.edu.cn
* extras: mirrors.btte.net
* updates: mirrors.btte.net
Setting up Install Process
Package wget-1.12-1.11.el6_5.x86_64 already installed and latest version
Nothing to do
adding EPEL repository
Running command: wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
--2014-06-07 06:32:38--http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Resolving dl.fedoraproject.org... 209.132.181.24, 209.132.181.25, 209.132.181.26, ...
Connecting to dl.fedoraproject.org|209.132.181.24|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14540 (14K)
Saving to: `epel-release-6-8.noarch.rpm.1'
0K .......... .... 100%359K=0.04s
2014-06-07 06:32:39 (359 KB/s) - `epel-release-6-8.noarch.rpm.1' saved
Running command: rpm -Uvh --replacepkgs epel-release-6*.rpm
Preparing... ##################################################
epel-release ##################################################
Running command: rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
Running command: rpm -Uvh --replacepkgs http://ceph.com/rpm-firefly/el6/noarch/ceph-release-1-0.el6.noarch.rpm
Retrieving http://ceph.com/rpm-firefly/el6/noarch/ceph-release-1-0.el6.noarch.rpm
Preparing... ##################################################
ceph-release ##################################################
Running command: yum -y -q install ceph
Package ceph-0.80.1-2.el6.x86_64 already installed and latest version
Running command: ceph --version
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
把秘钥及配置文件拷贝到客户端
# ceph-deploy admin ceph-client
found configuration file at: /root/.cephdeploy.conf
Invoked (1.5.3): /usr/bin/ceph-deploy admin ceph-client
Pushing admin keys and conf to ceph-client
connected to host: ceph-client
detect platform information from remote host
detect machine type
get remote short hostname
write cluster configuration to /etc/ceph/{cluster}.conf
正常centos6.4的系统是没有Module rbd的，在进行下面的操作时会出现报错：
#rbd map test-1 -p test --name client.admin-m 10.240.240.211 -k /etc/ceph/ceph.client.admin.keyring
ERROR: modinfo: could not find module rbd
FATAL: Module rbd not found.
rbd: modprobe rbd failed! (256)
解决的方法是升级内核版本：
Once you have deployed the almighty CEPH storage, you will want to be able to actualy use it (RBD).
Before we begin, some notes:
Current CEPH version: 0.67 (“dumpling”).
OS: Centos 6.4 x86_64 (running some VMs on KVM, basic CentOS qemu packages, nothing custom)
Since CEPH RBD module was first introduced with kernel 2.6.34 (and current RHEL/CentOS kernel is 2.6.32) – that means we need a newer kernel.
So, one of the options for the new kernel is, 3.x from elrepo.org:
rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml # will install 3.11.latest, stable, mainline
If you want that new kernel to boot by default, edit /etc/grub.conf, and change the Default=1 to Default=0, and reboot.
14、在客户端上应用ceph块存储
新建一个ceph pool
# rados mkpool test
在pool中新建一个镜像
# rbd create test-1 --size 4096 -p test -m 10.240.240.211 -k /etc/ceph/ceph.client.admin.keyring （“-m 10.240.240.211 -k /etc/ceph/ceph.client.admin.keyring”可以不用加）
把镜像映射到pool块设备中
#rbd map test-1 -p test --name client.admin-m 10.240.240.211 -k /etc/ceph/ceph.client.admin.keyring（“-m 10.240.240.211 -k /etc/ceph/ceph.client.admin.keyring”可以不用加）
查看rbd的映射关系
# rbd showmapped
id pool image snap device
0rbd foo - /dev/rbd0
1test test-1 - /dev/rbd1
2jiayuan jiayuan-img - /dev/rbd2
3jiayuan zhanguo - /dev/rbd3
4jiayuan zhanguo-5G- /dev/rbd4
把新建的镜像ceph块进行格式化
# mkfs.ext4 -m0 /dev/rbd1
新建一个挂载目录
# mkdir /mnt/ceph-rbd-test-1
把新建的镜像ceph块挂载到挂载目录
# mount /dev/rbd1 /mnt/ceph-rbd-test-1/
查看挂载情况
# df -h
Filesystem SizeUsed Avail Use% Mounted on
/dev/sda2 19G2.5G 15G15% /
tmpfs 116M 72K116M 1% /dev/shm
/dev/sda1 283M 52M213M20% /boot
/dev/rbd1 3.9G8.0M3.8G 1% /mnt/ceph-rbd-test-1
完成上面的步骤就可以向新建的ceph文件系统中存数据了。
15、在客户端上建立cephFS文件系统
# mkdir /mnt/mycephfs
# mount-t ceph 10.240.240.211:6789:/ /mnt/mycephfs -v -o name=admin,secret=AQDT9pNTSFD6NRAAoZkAgx21uGQ+DM/k0rzxow==
10.240.240.211:6789:/ on /mnt/mycephfs type ceph (rw,name=admin,secret=AQDT9pNTSFD6NRAAoZkAgx21uGQ+DM/k0rzxow==)
#上述命令中的name和secret参数值来自monitor的/etc/ceph/keyring文件：
# cat /etc/ceph/ceph.client.admin.keyring
key = AQDT9pNTSFD6NRAAoZkAgx21uGQ+DM/k0rzxow==
页:
[1]