SUSE11 Oracle 11g RAC双机集群环境部署

4321ddd

一. 安装前准备
(1) Linux系统版本
SUSE Linux Enterprise Server 11 (x86_64)
(2) Oracle database和Grid安装包
linux.x64_11gR2_database_1of2.zip
linux.x64_11gR2_database_2of2.zip
linux.x64_11gR2_grid.zip
(3) ASMlib安装包
oracleasm-support-2.1.8-1.SLE11.x86_64.rpm
oracleasmlib-2.0.4-1.sle11.x86_64.rpm
oracleasm-2.0.5-7.37.3.x86_64.rpm
oracleasm-kmp-default-2.0.5_3.0.76_0.11-7.37.3.x86_64.rpm
(4) 依赖软件包

gcc gcc-c++ gcc-32bit glibc-devel glibc-devel-32bit libaio libaio-devel libaio-devel-32bit libstdc++43-devel-32bit libstdc++43-devel sysstat libstdc++-devel libcap1 libcap1-32bit libcap2 libcap2-32bit compat* libgomp unixODBC unixODBC-devel

(5) 磁盘分区

名称	分区	容量
/	/dev/sda	100G
Swap	Swap	4G
CRS1	/dev/sdb1	2G
CRS2	/dev/sdb2	2G
CRS3	/dev/sdb3	2G
DATA1	/dev/sdb5	100G
REC1	/dev/sdb6	100G

(6) IP地址划分
  其中vip和scan ip不需要手动配置

Hostname	Node name	IP	Device	Type
hmracdb1	hmracdb1	192.168.6.154	eth1	Public
	hmracdb1-vip	192.168.6.54	eth1	VIP
	hmracdb1-priv	10.17.81.154	eth0	Private
hmracdb2	hmracdb2	192.168.6.155	eth1	Public
	hmracdb2-vip	192.168.6.55	eth1	VIP
	hmracdb2-priv	10.17.81.155	eth0	Private
Rac-scan	hmracdb-scan	192.168.6.66

二. 系统环境配置
(1) 主机hosts文件解析配置，两台主机使用相同配置

# public ip 192.168.6.154    hmracdb1 192.168.6.155    hmracdb2 # private ip 10.17.81.154     hmracdb1-priv 10.17.81.155     hmracdb2-priv # vip 192.168.6.54     hmracdb1-vip 192.168.6.55     hmracdb2-vip # scan ip 192.168.6.66     hmracdb-scan

(2) 创建oracle、grig用户和组

1. 创建组和用户 /usr/sbin/groupadd -g 501 oinstall /usr/sbin/groupadd -g 502 dba /usr/sbin/groupadd -g 503 oper /usr/sbin/groupadd -g 504 asmadmin /usr/sbin/groupadd -g 505 asmdba /usr/sbin/groupadd -g 506 asmoper /usr/sbin/useradd -u 501 -g oinstall -Gdba,oper,asmdba,asmadmin oracle -m /usr/sbin/useradd -u 502 -g oinstall -Gdba,asmadmin,asmdba,asmoper,oper grid -m 2. 创建密码 echo oracle | passwd --stdin oracle echo oracle | passwd --stdin grid # id oracle uid=501(oracle)gid=501(oinstall)groups=502(dba),503(oper),504(asmadmin),505(asmdba),501(oinstall) # id grid uid=502(grid) gid=501(oinstall)groups=502(dba),503(oper),504(asmadmin),505(asmdba),506(asmoper),501(oinstall)

(3) 创建安装目录

mkdir -p /u01/app/{grid,oracle} chown -R grid:oinstall /u01/ chown -R grid:oinstall /u01/app/grid/ chown -R oracle:oinstall /u01/app/oracle/ chmod -R 775 /u01/ ls -l /u01/app/ total 8 drwxrwxr-x 2 grid   oinstall 4096 Nov 16 19:09 grid drwxrwxr-x 2 oracle oinstall 4096 Nov 16 19:09 oracle

(4) 配置grid、oracle用户环境变量

------------------------------------------------------------------------------------------------------- 1. 配置hmracdb1节点grid用户的profile，ORACLE_SID为+ASM1，hmracdb2节点改为+ASM2 ------------------------------------------------------------------------------------------------------- export ORACLE_SID=+ASM1 export ORACLE_BASE=/u01/app/oracle exportORACLE_HOME=/u01/app/grid/product/11.2.0 export PATH=$PATH:$ORACLE_HOME/bin export TMP=/tmp export TMPDIR=$TMP export NLS_DATE_FORMAT="yyyy-mm-ddHH24:MI:SS" export THREADS_FLAG=native export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib export CVUQDISK GRP=oinstall if [ $USER = "oracle" ] || [$USER = 'grid' ];then    if [ $SHELL = "/bin/ksh" ];then        ulimit -p 16384        ulimit -n 65536    else        ulimit -u 16384 -n 65536    fi    umask 022 fi ------------------------------------------------------------------------------------------------------- 2. 配置hmracdb1节点oracle用户的profile，ORACLE_SID为rac1，hmracdb2节点改为rac2 ------------------------------------------------------------------------------------------------------- export ORACLE_SID=rac1 export ORACLE_BASE=/u01/app/oracle exportORACLE_HOME=$ORACLE_BASE/product/11.2.0 exportPATH=$ORACLE_HOME/bin:/usr/sbin:$PATH export TMP=/tmp export TMPDIR=$TMP export ORACLE_TERM=xterm exportLD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:/usr/lib64 exportCLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib export NLS_DATE_FORMAT="yyyy-mm-ddHH24:MI:SS" export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK if [ $USER = "oracle" ] || [$USER = 'grid' ];then    if [ $SHELL = "/bin/ksh" ];then        ulimit -p 16384        ulimit -n 65536    else        ulimit -u 16384 -n 65536    fi    umask 022 fi

(5) 修改内核参数配置

1. 修改/etc/sysctl.conf配置 # vim /etc/sysctl.conf kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 net.ipv4.ip_local_port_range = 9000 65500 fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.sem = 250 32000 100 128 kernel.shmmni = 4096 # sysctl –p 2. 修改/etc/security/limits.conf # vim /etc/security/limits.conf oracle          soft    nproc           2047 oracle          hard    nproc          16384 oracle          soft    nofile          1024 oracle          hard    nofile          65536 grid            soft    nproc           2047 grid            hard    nproc           16384 grid            soft    nofile          1024 grid            hard   nofile          65536 # vim /etc/pam.d/login session  required       pam_limits.so

(5) 配置NTP时间同步

1. 同步时间 # sntp -P no -r 10.10.0.2 &&hwclock –w 2. 配置hmracdb1为上层NTP服务器 hmracdb1:~ # vim /etc/ntp.conf    #添加以下配置 restrict hmracdb1 mask 255.255.255.0 nomodify notrap noquery server hmracdb1 server 127.127.1.0 3. 配置hmracdb2节点NTP服务 hmracdb2:~ # vim /etc/ntp.conf server hmracdb1 4. 两个节点修改如下配置，并重启ntp服务 # vim /etc/sysconfig/ntp   #修改以下配置 NTPD_OPTIONS="-x -g -u ntp:ntp" # service ntp restart

(6) 配置oracle、grid用户ssh密钥

su - grid grid@hmracdb1:~> ssh-keygen -t rsa grid@hmracdb2:~> ssh-keygen -t rsa grid@hmracdb1:~> ssh-copy-id -i~/.ssh/id_rsa.pub grid@hmracdb1 grid@hmracdb1:~> ssh-copy-id -i~/.ssh/id_rsa.pub grid@hmracdb2 grid@hmracdb2:~> ssh-copy-id -i~/.ssh/id_rsa.pub grid@hmracdb1 grid@hmracdb2:~> ssh-copy-id -i~/.ssh/id_rsa.pub grid@hmracdb2 su - oracle oracle@hmracdb1:~> ssh-keygen -t rsa oracle@hmracdb2:~> ssh-keygen -t rsa oracle@hmracdb1:~> ssh-copy-id -i~/.ssh/id_rsa.pub oracle@hmracdb1 oracle@hmracdb1:~> ssh-copy-id -i~/.ssh/id_rsa.pub oracle@hmracdb2 oracle@hmracdb2:~> ssh-copy-id -i~/.ssh/id_rsa.pub oracle@hmracdb1 oracle@hmracdb2:~> ssh-copy-id -i ~/.ssh/id_rsa.pub oracle@hmracdb2

三. 配置ASM磁盘
(1 )安装ASM软件(两个节点都安装)

# zypper in -y oracleasmoracleasm-kmp-default # zypper inoracleasmlib-2.0.4-1.sle11.x86_64.rpm # zypper in oracleasm-support-2.1.8-1.SLE11.x86_64.rpm

(2) 创建ASM分区(只需在节点1上创建)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104

hmracdb1:~ # fdisk /dev/sdb Device contains neither a valid DOSpartition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with diskidentifier 0xd60a0f97. Changes will remain in memory only, untilyou decide to write them. After that, of course, the previouscontent won't be recoverable. Warning: invalid flag 0x0000 of partitiontable 4 will be corrected by w(rite) Command (m for help): n Command action   e   extended   p   primary partition (1-4) p Partition number (1-4, default 1): Using default value 1 First sector (2048-419430399, default2048): Using default value 2048 Last sector, +sectors or +size{K,M,G}(2048-419430399, default 419430399): +2G Command (m for help): n Command action   e   extended   p   primary partition (1-4) p Partition number (1-4, default 2): Using default value 2 First sector (4196352-419430399, default4196352): Using default value 4196352 Last sector, +sectors or +size{K,M,G}(4196352-419430399, default 419430399): +2G Command (m for help): n Command action   e   extended   p   primary partition (1-4) p Partition number (1-4, default 3): Using default value 3 First sector (8390656-419430399, default8390656): Using default value 8390656 Last sector, +sectors or +size{K,M,G}(8390656-419430399, default 419430399): +2G Command (m for help): n Command action   e   extended   p   primary partition (1-4) e Selected partition 4 First sector (12584960-419430399, default12584960): Using default value 12584960 Last sector, +sectors or +size{K,M,G}(12584960-419430399, default 419430399): Using default value 419430399 Command (m for help): n First sector (12587008-419430399, default12587008): Using default value 12587008 Last sector, +sectors or +size{K,M,G}(12587008-419430399, default 419430399): +100G Command (m for help): n First sector (222304256-419430399,default 222304256): Using default value 222304256 Last sector, +sectors or +size{K,M,G}(222304256-419430399, default 419430399): Using default value 419430399 Command (m for help): p Disk /dev/sdb: 214.7 GB, 214748364800bytes 255 heads, 63 sectors/track, 26108cylinders, total 419430400 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes/ 512 bytes I/O size (minimum/optimal): 512 bytes /512 bytes Disk identifier: 0xd60a0f97   Device Boot      Start         End     Blocks   Id  System /dev/sdb1            2048     4196351    2097152   83  Linux /dev/sdb2         4196352     8390655    2097152   83  Linux /dev/sdb3         8390656    12584959    2097152   83  Linux /dev/sdb4        12584960   419430399  203422720    5  Extended /dev/sdb5        12587008   222302207  104857600   83  Linux /dev/sdb6       222304256   419430398   98563071+  83  Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. 在节点2上查看： hmracdb2:/usr/local/src # fdisk -l /dev/sdb Disk /dev/sdb: 214.7 GB, 214748364800bytes 255 heads, 63 sectors/track, 26108cylinders, total 419430400 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes/ 512 bytes I/O size (minimum/optimal): 512 bytes /512 bytes Disk identifier: 0xd60a0f97   Device Boot      Start         End      Blocks  Id  System /dev/sdb1            2048     4196351    2097152   83  Linux /dev/sdb2         4196352     8390655    2097152   83  Linux /dev/sdb3         8390656    12584959    2097152   83  Linux /dev/sdb4        12584960   419430399  203422720    5  Extended /dev/sdb5        12587008   222302207  104857600   83  Linux /dev/sdb6       222304256  419430398    98563071+  83 Linux

(3) 初始化ASM配置(两个节点都执行)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

1.加载asm内核模块 hmracdb1:~ # oracleasminit Creating /dev/oracleasm mount point:/dev/oracleasm Loading module "oracleasm":oracleasm Configuring "oracleasm" to usedevice physical block size Mounting ASMlib driver filesystem:/dev/oracleasm 2.初始化配置 hmracdb1:~ # /etc/init.d/oracleasmconfigure Configuring the Oracle ASM librarydriver. This will configure the on-bootproperties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions itwill have.  The current values will be shown in brackets ('[]').  Hitting <ENTER> without typing an answer will keep that current value.  Ctrl-C will abort. Default user to own the driver interface[]: grid Default group to own the driver interface[]: asmadmin Start Oracle ASM library driver on boot(y/n) [n]: y Scan for Oracle ASM disks on boot (y/n)[y]: y Writing Oracle ASM library driverconfiguration: done Initializing the Oracle ASMLibdriver:                                                                       done Scanning the system for Oracle ASMLib disks:                                                                 done

(4) 创建ASM盘(只需在节点1上创建)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

1. 在节点1上ASM磁盘 hmracdb1:~ # oracleasm listdisks hmracdb1:~ # oracleasm createdisk CRS1/dev/sdb1 Writing disk header: done Instantiating disk: done hmracdb1:~ # oracleasm createdisk CRS2/dev/sdb2 Writing disk header: done Instantiating disk: done hmracdb1:~ # oracleasm createdisk CRS3/dev/sdb3 Writing disk header: done Instantiating disk: done hmracdb1:~ # oracleasm createdisk DATA1/dev/sdb5 Writing disk header: done Instantiating disk: done hmracdb1:~ # oracleasm createdisk REC1/dev/sdb6 Writing disk header: done Instantiating disk: done hmracdb1:~ # oracleasm listdisks CRS1 CRS2 CRS3 DATA1 REC1 2. 节点2上扫描ASM共享盘 hmracdb2:~ # oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "CRS1" Instantiating disk "CRS2" Instantiating disk "CRS3" Instantiating disk "DATA1" Instantiating disk "REC1" hmracdb2:~ # oracleasm listdisks CRS1 CRS2 CRS3 DATA1 REC1 3. 检查ASM磁盘路径 hmracdb2:~ # oracleasm querydisk/dev/sdb* Device "/dev/sdb" is not markedas an ASM disk Device "/dev/sdb1" is marked anASM disk with the label "CRS1" Device "/dev/sdb2" is marked anASM disk with the label "CRS2" Device "/dev/sdb3" is marked anASM disk with the label "CRS3" Device "/dev/sdb4" is notmarked as an ASM disk Device "/dev/sdb5" is marked anASM disk with the label "DATA1" Device "/dev/sdb6" is marked anASM disk with the label "REC1" hmracdb2:~ # ll /dev/oracleasm/disks/ total 0 brw-rw---- 1 grid asmadmin 8, 17 Nov 2114:19 CRS1 brw-rw---- 1 grid asmadmin 8, 18 Nov 2114:19 CRS2 brw-rw---- 1 grid asmadmin 8, 19 Nov 2114:19 CRS3 brw-rw---- 1 grid asmadmin 8, 21 Nov 2114:19 DATA1 brw-rw---- 1 grid asmadmin 8, 22 Nov 2114:19 REC1 hmracdb2:~ # ll -ltr /dev | grep "8,*17" brw-rw---- 1 root disk      8, 17 Nov 21 14:19 sdb1 hmracdb2:~ # ll -ltr /dev | grep "8,*18" brw-rw---- 1 root disk      8, 18 Nov 21 14:19 sdb2 hmracdb2:~ # ll -ltr /dev | grep "8,*19" brw-rw---- 1 root disk      8, 19 Nov 21 14:19 sdb3 hmracdb2:~ # ll -ltr /dev | grep "8,*21" brw-rw---- 1 root disk      8,  21 Nov 21 14:19 sdb5 hmracdb2:~ # ll -ltr /dev | grep "8,*22" brw-rw---- 1 root disk      8, 22 Nov 21 14:19 sdb6

(5) 安装oracle依赖软件包

hmracdb1:~ # for i in "gcc gcc-c++gcc-32bit glibc-devel glibc-devel-32bit libaio libaio-devel libaio-devel-32bitlibstdc++43-devel-32bit libstdc++43-devel sysstat libstdc++-devel libcap1libcap1-32bit libcap2 libcap2-32bit compat* libgomp unixODBC unixODBC-devel";dorpm -q $i;done hmracdb1:~ # zypper in gcc gcc-c++ gcc-32bit glibc-develglibc-devel-32bit libaio libaio-devel libaio-devel-32bitlibstdc++43-devel-32bit libstdc++43-devel sysstat libstdc++-devel libcap1libcap1-32bit libcap2 libcap2-32bit compat* libgomp unixODBC unixODBC-devel

(6) 安装cvuqdisk软件

解压grid软件包，安装grid目录里的cvuqdisk-1.0.7-1.rpm软件 hmracdb1:/usr/local/src # cd grid/rpm/ hmracdb1:/usr/local/src/grid/rpm # ls cvuqdisk-1.0.7-1.rpm hmracdb1:/usr/local/src/grid/rpm # rpm -ivh cvuqdisk-1.0.7-1.rpm

(7) 配置nslookup

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

1. 节点1上配置 hmracdb1:~ # mv /usr/bin/nslookup/usr/bin/nslookup.original hmracdb1:~ # vim /usr/bin/nslookup #!/bin/bash HOSTNAME=${1} if [[ $HOSTNAME ="hmracdb-scan" ]]; then    echo "Server: 192.168.6.54 "    echo "Address: 192.168.6.54 #53"    echo "Non-authoritative answer:"    echo "Name: hmracdb-scan"    echo "Address: 192.168.6.66 " else    /usr/bin/nslookup.original $HOSTNAME fi hmracdb1:~ # chmod 755 /usr/bin/nslookup 2. 节点2上配置 hmracdb2:~ # mv /usr/bin/nslookup/usr/bin/nslookup.original hmracdb2:~ # vim /usr/bin/nslookup #!/bin/bash HOSTNAME=${1} if [[ $HOSTNAME ="hmracdb-scan" ]]; then    echo "Server: 192.168.6.55 "    echo "Address: 192.168.6.55 #53"    echo "Non-authoritative answer:"     echo "Name: hmracdb-scan"    echo "Address: 192.168.6.66 " else    /usr/bin/nslookup.original $HOSTNAME fi hmracdb2:~ # chmod 755 /usr/bin/nslookup 3. 测试 hmracdb1:~ # nslookup hmracdb-scan Server: 192.168.6.54 Address: 192.168.6.54 #53 Non-authoritative answer: Name: hmracdb-scan Address: 192.168.6.66

(8) Gird安装先决条件检查

1. 修改属主权限为grid用户 hmracdb1:/usr/local/src # chown -R grid.grid 2. 以grid用户执行安装先决条件检查 grid@hmracdb1:/usr/local/src/grid>./runcluvfy.sh stage -pre crsinst -n hmracdb1,hmracdb2 -fixup -verbose

四. 图形化界面安装Grid Infrastructure
安装方法：

方法1：使用VNC软件登陆服务器图形界面安装

方法2：使用Xmanager调用图形界面安装

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

ln -s /usr/bin/env /bin/ grid@hmracdb1:/usr/local/src/grid>export DISPLAY=10.18.221.155:0.0 grid@hmracdb1:/usr/local/src/grid>xhost + access control disabled, clients canconnect from any host grid@hmracdb1:/usr/local/src/grid>./runInstaller Starting Oracle Universal Installer... Checking Temp space: must be greater than120 MB.   Actual 83913 MB    Passed Checking swap space: must be greater than150 MB.   Actual 4094 MB    Passed Checking monitor: must be configured todisplay at least 256 colors.    Actual16777216    Passed Preparing to launch Oracle UniversalInstaller from /tmp/OraInstall2016-11-22_09-16-06AM. Please wait...grid@hmracdb1:/usr/local/src/grid> grid@hmracdb1:/usr/local/src/grid>

暴走小骚年

感谢分享啊