|
实验说明:此处只用一个SAN设备与一个WWW1节点做实验步骤
www1 eth0:192.168.1.1
SAN eth0:192.168.1.2 添加5块为4G的硬盘
实验步骤:
5.1 iscsi服务器上所做的配置
1、在SAN服务器上添加5块硬盘
配置eth0网卡
[iyunv@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
# Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
DEVICE=eth0
BOOTPROTO=static
HWADDR=00:0C:29:4C:BD:42
ONBOOT=yes
IPADDR=192.168.1.1
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
[iyunv@localhost ~]# service network restart
首先使用“fdisk”命令在每块硬盘上创建一个分区
[iyunv@localhost ~]# fdisk /dev/sdb
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n //创建
Command action
e extended
p primary partition (1-4)
p //分区
Partition number (1-4): 1 //第一块
First cylinder (1-522, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-522, default 522):
Using default value 522
Command (m for help): t //类型
Selected partition 1
Hex code (type L to list codes): fd //raid类型
Changed system type of partition 1 to fd (Linux raid autodetect)
Command (m for help): p //查看
Disk /dev/sdb: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 522 4192933+ fd Linux raid autodetect
Command (m for help): w //保存
[iyunv@localhost ~]# fdisk /dev/sdc
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n //创建
Command action
e extended
p primary partition (1-4)
p //分区
Partition number (1-4): 1 //第一块
First cylinder (1-522, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-522, default 522):
Using default value 522
Command (m for help): t //类型
Selected partition 1
Hex code (type L to list codes): fd //raid类型
Changed system type of partition 1 to fd (Linux raid autodetect)
Command (m for help): p //查看
Disk /dev/sdb: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 522 4192933+ fd Linux raid autodetect
Command (m for help): w //保存
[iyunv@localhost ~]# fdisk /dev/sdd
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n //创建
Command action
e extended
p primary partition (1-4)
p //分区
Partition number (1-4): 1 //第一块
First cylinder (1-522, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-522, default 522):
Using default value 522
Command (m for help): t //类型
Selected partition 1
Hex code (type L to list codes): fd //raid类型
Changed system type of partition 1 to fd (Linux raid autodetect)
Command (m for help): p //查看
Disk /dev/sdb: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 522 4192933+ fd Linux raid autodetect
Command (m for help): w //保存
[iyunv@localhost ~]# fdisk /dev/sde
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n //创建
Command action
e extended
p primary partition (1-4)
p //分区
Partition number (1-4): 1 //第一块
First cylinder (1-522, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-522, default 522):
Using default value 522
Command (m for help): t //类型
Selected partition 1
Hex code (type L to list codes): fd //raid类型
Changed system type of partition 1 to fd (Linux raid autodetect)
Command (m for help): p //查看
Disk /dev/sdb: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 522 4192933+ fd Linux raid autodetect
Command (m for help): w //保存
[iyunv@localhost ~]# fdisk /dev/sdf
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n //创建
Command action
e extended
p primary partition (1-4)
p //分区
Partition number (1-4): 1 //第一块
First cylinder (1-522, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-522, default 522):
Using default value 522
Command (m for help): t //类型
Selected partition 1
Hex code (type L to list codes): fd //raid类型
Changed system type of partition 1 to fd (Linux raid autodetect)
Command (m for help): p //查看
Disk /dev/sdb: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 522 4192933+ fd Linux raid autodetect
Command (m for help): w //保存
创建软raid 磁盘阵列
[iyunv@localhost ~]# mdadm -Cv /dev/md5 -a yes -n4 -l5 -x1 /dev/sd[b-f]1 //创建软raid,-a:对应设备不存在责自动创建,-n:使用几块硬盘,-l:使用的raid级别是5,-x:有几块硬盘作为备份用
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: size set to 4192832K
mdadm: array /dev/md5 started.
将创建好的raid应用到逻辑卷
[iyunv@localhost ~]# pvcreate /dev/md5 //创建物理卷
Physical volume "/dev/md5" successfully created
[iyunv@localhost ~]# vgcreate vol0 /dev/md5 //创建卷组
/dev/cdrom: open failed: 只读文件系统
/dev/cdrom: open failed: 只读文件系统
Attempt to close device '/dev/cdrom' which is not open.
/dev/cdrom: open failed: 只读文件系统
Attempt to close device '/dev/cdrom' which is not open.
/dev/cdrom: open failed: 只读文件系统
Attempt to close device '/dev/cdrom' which is not open.
Volume group "vol0" successfully created
[iyunv@localhost ~]# lvcreate -L 4G -n lv-host1 vol0 //创建逻辑卷,-L:指定创建磁盘大小,-n:新创建的逻辑卷名字,vol0:卷组名字
/dev/cdrom: open failed: 只读文件系统
Logical volume "lv-host1" created
[iyunv@localhost ~]# lvcreate -L 4G -n lv-host2 vol0
/dev/cdrom: open failed: 只读文件系统
Logical volume "lv-host2" created
[iyunv@localhost ~]# lvdisplay /dev/vol0/lv-host1 /dev/vol0/lv-host2 //查看创建成功的逻辑卷
为了重建或恢复软raid方便,将此阵列配置保存为/etc/mdadm.conf,此文件默认不错在,手动创建
[iyunv@localhost ~]# mdadm -D -s > /etc/mdadm.conf
[iyunv@localhost ~]# vi /etc/mdadm.conf
ARRAY /dev/md5 level=raid5 num-devices=4 metadata=0.90 spares=1 UUID=325a6875:fe114112:ef89899e:2459e80b auto=yes //开机自动加载软raid
安装iscsi服务器也称为Target端,负责定义存储对象,安装相关工具,对应的系统服务为tgtd,命令行管理工具为tgtadm
[iyunv@localhost ~]# mount /dev/cdrom /media/
mount: block device /dev/cdrom is write-protected, mounting read-only
[iyunv@localhost ~]# vim /etc/yum.repos.d/rhel-debuginfo.repo //编辑下列两处
[rhel-debuginfo]
name=Red Hat Enterprise Linux $releasever - $basearch - Debug
baseurl=file:///media/ClusterStorage
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[rhel-debuginfo-beta]
name=Red Hat Enterprise Linux $releasever Beta - $basearch - Debug
baseurl=file:///media/Server
enabled=1
gpgcheck=0
[iyunv@localhost ~]# yum -y install scsi-target-utils //安装iscsi定义工具
2、创建iscsi对象(target)
[iyunv@localhost ~]# service tgtd start //启动tgtd服务
Starting SCSI target daemon: Starting target framework daemon
[iyunv@localhost ~]# netstat -anpl |grep tgtd //查看tgtd端口
tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 5501/tgtd
tcp 0 0 :::3260 :::* LISTEN 5501/tgtd
unix 2 [ ACC ] STREAM LISTENING 18142 5501/tgtd /tmp/.TGT_IPC_ABSTRACT_NAMESPACE
unix 2 [ ] DGRAM 18135 5501/tgtd
创建两个iscsi对象
[iyunv@localhost ~]# tgtadm -L iscsi -o new -m target -t 1 -T iqn.2011-5-11:lvm1 //-L:设备类型,-o:目标动作,是drop、new、delete,-m:管理目标(target服务器,logicaalunit客户机)-t:创建设备的ID号,-T:创建设备的名字
[iyunv@localhost ~]# tgtadm -L iscsi -o new -m target -t 2 -T iqn.2011-5-11:lvm2 //创建一个ID号为2,名字为iqn.2011-5-11的iscsi存储设备
[iyunv@localhost ~]# tgtadm -L iscsi -o delete -m target -t 3 //删除一个iscsi设备
[iyunv@localhost ~]# tgtadm -L iscsi -o show -m target //查看本机创建的iscsi设备
Target 1: iqn.2011-5-11:lvm1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: None
Account information:
ACL information:
Target 2: iqn.2011-5-11:lvm2
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00020000
SCSI SN: beaf20
Size: 0 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: None
Account information:
ACL information:
3、为客户机分配存储空间(LUN)
将逻辑卷lv-host1和lv-host2分别挂在到tid为1、2的target下,指派给客户机192.168.1.0/24网段的客户机使用,
[iyunv@localhost ~]# tgtadm -L iscsi -o new -m logicalunit -t 1 -l 1 -b /dev/vol0/lv-host1 //-l:指定LUN的ID号,表示第一个LUN。-b:指定设备位置
[iyunv@localhost ~]# tgtadm -L iscsi -o new -m logicalunit -t 2 -l 1 -b /dev/vol0/lv-host2 //表示将/dev/vol0/lv-host1分区应用到tid为2的iscsi(target服务器)上,-m logicalunit:表示管理目标为LUN,-l:表示在tid号为1上创建的第一个LUN
[iyunv@localhost ~]# tgtadm -L iscsi -o bind -m target -t 1 -I 192.168.1.0/24 //允许哪些客户机使用这个tid号为1的iscsi设备。-o bind:绑定客户机地址,
[iyunv@localhost ~]# tgtadm -L iscsi -o bind -m target -t 2 -I 192.168.1.0/24
[iyunv@localhost ~]# history |tail -12 >> /etc/rc.local //将target和LUN配置追加到/etc/rc.local开机自动执行此脚本
[iyunv@localhost ~]# vim /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
tgtadm -L iscsi -o new -m target -t 1 -T iqn.2011-5-11:lvm1 //删除其余行,只留此6行
tgtadm -L iscsi -o new -m target -t 2 -T iqn.2011-5-11:lvm2
tgtadm -L iscsi -o new -m logicalunit -t 1 -l 1 -b /dev/vol0/lv-host1
tgtadm -L iscsi -o new -m logicalunit -t 2 -l 1 -b /dev/vol0/lv-host2
tgtadm -L iscsi -o bind -m target -t 1 -I 192.168.1.0/24
tgtadm -L iscsi -o bind -m target -t 2 -I 192.168.1.0/24
[iyunv@localhost ~]# tgtadm -L iscsi -o show -m target //查看target,发现LUN类型为disk
Target 1: iqn.2011-5-11:lvm1
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: None
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 4295 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /dev/vol0/lv-host1
Account information:
ACL information:
192.168.1.0/24
Target 2: iqn.2011-5-11:lvm2
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00020000
SCSI SN: beaf20
Size: 0 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: None
LUN: 1
Type: disk
SCSI ID: IET 00020001
SCSI SN: beaf21
Size: 4295 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /dev/vol0/lv-host2
Account information:
ACL information:
192.168.1.0/24
4、保存target和LUN配置
修改/etc/tgt/targets.conf配置文件,将target、LUN信息保存为静态配置,使重启tgtd服务后仍有效
[iyunv@localhost ~]# vim /etc/tgt/targets.conf
在文件最后添加如下行
<target iqn.2011-5-11:lvm1>
backing-store /dev/vol0/lv-host1
initiator-address 192.168.1.0/24
</target>
<target iqn.2011-5-11:lvm2>
backing-store /dev/vol0/lv-host2
initiator-address 192.168.1.0/24
</target>
[iyunv@localhost ~]# chkconfig tgtd on
5.2使用iscsi存储设备(客户机www1)节点
配置客户机eth0网卡
[iyunv@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
# Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
DEVICE=eth0
BOOTPROTO=static
HWADDR=00:0C:29:AB:B0:02
ONBOOT=yes
IPADDR=192.168.1.2
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
Iscsi客户机通常也称为initiator,负责连接iscsi设备,在iscsi上创建文件系统,以及存储数据。安装客户端所需工具
[iyunv@localhost ~]# mount /dev/cdrom /media/
mount: block device /dev/cdrom is write-protected, mounting read-only
[iyunv@localhost ~]# vim /etc/yum.repos.d/rhel-debuginfo.repo
[rhel-debuginfo]
name=Red Hat Enterprise Linux $releasever - $basearch - Debug
baseurl=file:///media/Server
enabled=1
gpgcheck=0
[iyunv@localhost ~]# yum -y install iscsi-initiator-utils lsscsi
1、发现并连接iscsi设置
使用iscsiadm工具之前,先启动iscsi服务。查找iscsi设备,其中 “–m discovery” 表示发现/查找,“-t sendtargets ”表示查询目标(iscsi设备),“-p ip地址”用来指定iscsi服务器的ip地址
[iyunv@localhost ~]# service iscsi restart //启动iscsi服务
Stopping iSCSI daemon: iscsiadm: can not connect to iSCSI daemon (111)!
iscsiadm: initiator reported error (20 - could not connect to iscsid)
iscsiadm: Could not stop iscsid. Trying sending iscsid SIGTERM or SIGKILL signals manually
iscsid 已停 [确定]
Starting iSCSI daemon: [确定]
[确定]
设置 iSCSI 目标:iscsiadm: No records found!
[确定]
[iyunv@localhost ~]# iscsiadm -m discovery -t sendtargets -p 192.168.1.1
192.168.1.1:3260,1 iqn.2011-5-11:lvm1
192.168.1.1:3260,1 iqn.2011-5-11:lvm2
[iyunv@localhost ~]# iscsiadm -m node -T iqn.2011-5-11:lvm1 --login //将上面查找到的地址复制下来即可
Logging in to [iface: default, target: iqn.2011-5-11:lvm1, portal: 192.168.1.1,3260]
Login to [iface: default, target: iqn.2011-5-11:lvm1, portal: 192.168.1.1,3260]: successful
[iyunv@localhost ~]# iscsiadm -m node -T iqn.2011-5-11:lvm2 --login //登陆iscsi成功
Logging in to [iface: default, target: iqn.2011-5-11:lvm2, portal: 192.168.1.1,3260]
Login to [iface: default, target: iqn.2011-5-11:lvm2, portal: 192.168.1.1,3260]: successful
[iyunv@localhost ~]# lsscsi //查看iscsi设备列表,发现已有(IET代表是iscsi设备)
[0:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda
[1:0:0:0] storage IET Controller 0001 -
[1:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdb
[2:0:0:0] storage IET Controller 0001 -
[2:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdc
2、使用iscsi存储设备
1)为磁盘分区
[iyunv@localhost ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 4294 MB, 4294967296 bytes
133 heads, 62 sectors/track, 1017 cylinders
Units = cylinders of 8246 * 512 = 4221952 bytes
Disk /dev/sdb doesn't contain a valid partition table //新的iscsi设备无分区列表
[iyunv@localhost ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1017, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1017, default 1017):
Using default value 1017
Command (m for help): p
Disk /dev/sdb: 4294 MB, 4294967296 bytes
133 heads, 62 sectors/track, 1017 cylinders
Units = cylinders of 8246 * 512 = 4221952 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 1017 4193060 83 Linux
Command (m for help): w
[iyunv@localhost ~]# fdisk /dev/sdc
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1017, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1017, default 1017):
Using default value 1017
Command (m for help): p
Disk /dev/sdc: 4294 MB, 4294967296 bytes
133 heads, 62 sectors/track, 1017 cylinders
Units = cylinders of 8246 * 512 = 4221952 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 1017 4193060 83 Linux
Command (m for help): w
2)创建文件系统(格式化)
[iyunv@localhost ~]# mkfs.ext3 /dev/sdb1
[iyunv@localhost ~]# mkfs.ext3 /dev/sdc1
3)挂载文件系统
[iyunv@localhost ~]# yum -y install http* mysql* //在www1上安装apahce、mysql
[iyunv@localhost ~]# service httpd restart
停止 httpd: [失败]
启动 httpd: [确定]
2、挂载分区
[iyunv@localhost ~]# mkdir /var/www/html/bbs/
[iyunv@localhost ~]# mkdir /var/lib/mysql/
[iyunv@localhost ~]# service mysqld start //启动MYSQL服务
[iyunv@localhost ~]# mysqladmin -uroot password 123 //设置mysql管理员密码
[iyunv@localhost ~]# mkdir /mysql/
[iyunv@localhost ~]# cp -rp /var/lib/mysql/* /mysql/
[iyunv@localhost ~]# rm -rf /var/lib/mysql/* //清空mysql下所有内容挂载
[iyunv@localhost ~]# mount /dev/sdb1 /var/www/html/bbs/
[iyunv@localhost ~]# mount /dev/sdc1 /var/lib/mysql/
[iyunv@localhost ~]# df -hT
文件系统 类型 容量 已用 可用 已用% 挂载点
/dev/sda5 ext3 17G 5.1G 11G 33% /
/dev/sda2 ext3 2.0G 36M 1.9G 2% /home
/dev/sda1 ext3 99M 12M 83M 13% /boot
tmpfs tmpfs 252M 0 252M 0% /dev/shm
/dev/sdb1 ext3 4.0G 87M 3.7G 3% /var/www/html/bbs
/dev/sdc1 ext3 4.0G 93M 3.7G 3% /var/lib/mysql
[iyunv@localhost html]# cd /var/www/html/bbs
[iyunv@localhost bbs]# ls
index.html lost+found //如出现此字,则挂载正确,/var/lib/mysql也是一样
3、保存iscsi连接配置
在某些客户机系统中,有时候会因为设备连接变动、引导顺序等因素影响,导致iscsi存储设备的设备文件异常(可能不在是/dev/sdb1或/dev/sdc1),可以通过系统的ID号,即UUID可进行定位
[iyunv@localhost ~]# chkconfig iscsi on
[iyunv@localhost ~]# blkid /dev/sdb1
/dev/sdb1: UUID="5e554c1b-bfdf-41fb-9420-691d286a6124" SEC_TYPE="ext2" TYPE="ext3"
[iyunv@localhost ~]# blkid /dev/sdc1
/dev/sdc1: UUID="e44f5d1c-a97f-4123-b286-b4708fad1d41" SEC_TYPE="ext2" TYPE="ext3"
[iyunv@localhost ~]# vim /etc/fstab
在文件最后添加如下行
UUID=5e554c1b-bfdf-41fb-9420-691d286a6124 /var/www/html/bbs ext3 defaults,_netdev 0 0
UUID=e44f5d1c-a97f-4123-b286-b4708fad1d41 /var/lib/mysql ext3 defaults,_netdev 0 0
[iyunv@localhost ~]#mount –a
[iyunv@localhost ~]# mount //查看已挂载
/dev/sda5 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sda2 on /home type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /var/www/html/bbs type ext3 (rw,_netdev)
/dev/sdc1 on /var/lib/mysql type ext3 (rw,_netdev)
上传Discuz_7.2_FULL_SC_UTF8.zip论坛软件到/root下
[iyunv@localhost ~]# unzip Discuz_7.2_FULL_SC_UTF8.zip -d discuz
[iyunv@localhost ~]# mv discuz/upload/* /var/www/html/bbs/
[iyunv@localhost ~]# cp -rp /mysql/* /var/lib/mysql/
编辑httpd.conf配置文件
[iyunv@localhost ~]# vim /etc/httpd/conf/httpd.conf
DocumentRoot "/var/www/html/bbs"
AddDefaultCharset gb2312
DirectoryIndex index.php index.html index.html.var
[iyunv@localhost ~]# service httpd restart
停止 httpd: [确定]
启动 httpd: [确定]
|
|