Storage Foundation+Oracle RAC集群部署.docx

1 数据库集群部署1.1 环境1.1.1 硬件环境配置128G内存4路8核心CPU的两台宝德服务器、一台或两台光纤交换机、存储设备、备份设备、千兆以太网络交换机、光纤线缆及网络电缆若干。1.1.2 软件环境操作系统：redhat 6.4 rhel-server-6.4-x86_64文件系统：storage_foundation_for_oracle_rac(6.1)数据库版本：Oracle Database Enterprise Edition for linux1.2 IP地址划分注意：网卡绑定要将不同物理网卡的端口进行绑定，如下所示eth0和eth1为同一物理网卡，eth2和eth3是同一物理网卡，故将eth0和eth2绑定作为公网IP，eth1和eth3作为私有IP。DNS区域命名是根据当前区域来配置，这里使用作为该区域，请在不同的现场使用不同的区域命名。l sfrac1：数据库服务器1l sfrac2：数据库服务器2l 主机名网卡IP地址备注sfrac1Bond0: eth0 eth224sfrac1. sfrac1N/A25sfrac1-vip. sfrac1Eth1sfrac1-priv1sfrac1Eth31sfrac1-priv2sfrac2Bond0: eth0 eth226sfrac2. sfrac2N/A27sfrac2-vip. sfrac2Eth2sfrac1-priv1sfrac2Eth32sfrac1-priv2sfrac-scanN/A21sfrac-scanN/A22sfrac-scanN/A23主机/etc/hosts文件内容，sfrac-scan使用DNS解析，故不需要添加。1.3 共享磁盘划分注意：表 1对应的磁盘设备都是在sfrac1上进行查询得到的（在做查询对比的时候，请在主节点sfrac1上进行查询配置，两节点的磁盘设备名可能不同）表 1 共享磁盘分布情况磁盘设备名Dg名Vol名磁盘大小挂载点/备注aluadisk0_4ocrvotedgocrvotevol3G/ocraluadisk0_1oradatadgoradatavol2.8T/oradataaluadisk0_0aluadisk0_2aluadisk0_3vxfendg2G*3Io/fen协调盘aluadisk0_7aluadisk0_8aluadisk0_9vxfendgbak1G*3Io/fen作为故障切换时使用aluadisk0_5ftpdgftpvol4.2T/ftpaluadisk0_10dbbakdgdbbakvol7T/dbbakaluadisk0_11ftpbakdgftpbakvol9T/ftpbakaluadisk0_12200GLinux系统备份，备份完成将其卸载aluadisk0_13oradatadgudoradatavolud3.2T/oradata 作为故障切换时使用aluadisk0_14ftpdgudftpvolud4.2T/ftp 作为故障切换时使用aluadisk0_14ocrvotedgudocrvotevolud20G/ocr 作为故障切换时使用1.4 Linux系统安装以下为分区情况，为了方便后期做系统备份，采用lvm卷管理。其中/boot分区500M，采用标准分区Swap分区64G，采用标准分区/分区200G，采用LVM/u01分区剩余空间290G，采用LVMrootsfrac1 oradata# /bin/df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/vg_rac-LogVol_root 197G 6.3G 181G 4% /tmpfs 16G 72K 16G 1% /dev/shm/dev/sda1 485M 38M 422M 9% /boot/dev/mapper/vg_rac-LogVol_u01 289G 8.7G 266G 4% /u011.5 系统环境1.5.1 网络配置 配置绑定网卡执行命令关闭NetworkManager服务，此服务可能造成绑定出现问题：service NetworkManager stop执行命令设置开机不启动：chkconfig NetworkManager off新建并编辑bond0虚拟网卡配置文件如下：rootsfrac1 # vim /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0BOOTPROTO=noneONBOOT=yesIPADDR=24NETMASK=USERCTL=noDNS1=51保存原配置文件，并编辑eth0网卡配置文件如下cp /etc/sysconfig/network-scripts/ifcfg-eth0/etc/sysconfig/network-scripts/ifcfg-eth0_bakrootsfrac1 # vim /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0TYPE=EthernetONBOOT=yesBOOTPROTO=noneMASTER=bond0SLAVE=yesUSERCTL=yes保存原配置文件,编辑eth2网卡配置文件如下：cp /etc/sysconfig/network-scripts/ifcfg-eth2 /etc/sysconfig/network-scripts/ifcfg-eth2_bakrootsfrac1 # vim /etc/sysconfig/network-scripts/ifcfg-eth2DEVICE=eth2TYPE=EthernetONBOOT=yesBOOTPROTO=noneMASTER=bond0SLAVE=yesUSERCTL=yes修改别名信息/etc/modprobe.d/dist.conf添加如下信息：alias bond0 bondingoptions bond0 mode=1 miimon=100修改启动项/etc/rc.local添加如下信息ifenslave bond0 eth0 eth1重启网络服务service network restart 配置DNS客户端修改解析配置文件如下：rootsfrac1 # vim /etc/resolv.conf nameserver 51search 配置NTP客户端执行命令同步系统时钟和硬件时钟：rootsfrac1 # clock systohc修改配置文件/etc/ntp.conf修改如下选项：server 51 preferserver 26修改配置文件/etc/sysconfig/ntpd添加如下配置（OPTION配置项添加-x）：rootsfrac1 # cat /etc/sysconfig/ntpd# Drop root to id ntp:ntp by default.SYNC_HWCLOCK=yesOPTIONS=-x -u ntp:ntp -p /var/run/ntpd.pid -g重启NTP服务service ntpd restartchkconfig ntpd on1.5.2 配置HOSTS解析文件每个节点vim /etc/hostsrootsfrac1 # vim /etc/hosts localhost localhost.localdomain localhost4 localhost4.localdomain4:1 localhost localhost.localdomain localhost6 localhost6.localdomain624sfrac126sfrac225sfrac1-vip27sfrac2-vip1sfrac1-priv1sfrac1-priv2sfrac2-priv2sfrac2-priv2sfrac-scan解析采用DNS服务器，不用写入hosts文件。1.5.3 yum安装依赖包每个节点执行配置yum源vi /etc/yum.repos.d/redhat.repoRHELname=RHEL6.4baseurl=file:/mediagpgcheck=0gpgkey=file:/media/RPM-GPG-KEY-redhat-releaseenabled=1插入RHEL6.4光盘，挂载光盘到/mediarootsfrac1 sysconfig# mount /dev/cdrom /mediamount: block device /dev/sr0 is write-protected, mounting read-only检查yum源是否配置正确rootsfrac1 sysconfig# yum list如果有如下输出，则配置正确yum安装依赖包yum install -y compat-libstdc+-33 elfutils-libelf elfutils-libelf-devel gcc gcc-c+ glibc glibc-common glibc-devel glibc-headers ksh libaio libaio-devel libgcc libstdc+ libstdc+-devel make sysstat unixODBC unixODBC-devel mksh nss-softokn-freebl-3.12.9-11.el6.i686 libgcc-4.4.7-3.el6.i686 libstdc+-4.4.7-3.el6.i686 pam-1.1.1-13.el6.i686 dumpxterm vsftp执行ln -s /lib64/libcap.so.2 /lib64/libcap.so.1防止安装GI时运行root.sh报错1.5.4 修改内核参数每个节点执行处级32G内存设置为：rootrac2 # vi /etc/sysctl.confkernel.shmall = 8388608 -此参数设置为系统内存/4096kernel.shmmax = 34359738367 -此参数设置成系统内存大小减1bytekernel.shmmni = 4096kernel.sem = 250 32000 100 128fs.file-max = 6815744net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 1048576net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048576fs.aio-max-nr = 1048576局级128G内存设置为：kernel.shmall = 16777216 -此参数设置为系统内存/4096kernel.shmmax = 68719476735 -此参数设置成系统内存大小减1bytekernel.shmmni = 4096kernel.sem = 250 32000 100 128fs.file-max = 6815744net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 1048576net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048576fs.aio-max-nr = 1048576执行以下命令立即生效rootrac1 sysconfig# sysctl -p1.5.5 修改IO计划每个节点修改menu.list配置文件rootrac2 # vim /boot/grub/menu.lstdefault=0timeout=5splashimage=(hd0,0)/grub/splash.xpm.gzhiddenmenutitle Red Hat Enterprise Linux (2.6.32-358.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/VolGroup-LogVol00rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=VolGroup/LogVol00 rd_NO_DM rhgb quiet initrd /initramfs-2.6.32-358.el6.x86_64.img修改如下default=0timeout=5splashimage=(hd0,0)/grub/splash.xpm.gzhiddenmenutitle Red Hat Enterprise Linux (2.6.32-358.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/VolGroup-LogVol00 elevator=deadline rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=VolGroup/LogVol00 rd_NO_DM rhgb quiet initrd /initramfs-2.6.32-358.el6.x86_64.img重启系统rootrac2 # rebootBroadcast message from rootrac2 (/dev/pts/0) at 10:41 .The system is going down for reboot NOW!1.5.6 配置安全设置等关闭防火墙，每个节点执行命令：service iptables stopservice ip6tables stopchkconfig iptables offchkconfig ip6tables off将selinux禁用即可，也就是修改/etc/selinux/config文件，修改后的内容为：SELINUX=disabled1.5.7 修改用户限制vi /etc/security/limits.conf oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 4096 oracle hard nofile 65536 oracle soft stack 10240 grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 4096 grid hard nofile 65536 grid soft stack 102401.5.8 修改环境变量每个节点执行vim /etc/profileif /$USER = oracle | /$USER = grid ; then if /$SHELL = /bin/ksh ; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi1.5.9 创建用户和设置用户变量分别在各个节点创建用户组groupadd -g 1000 oinstallgroupadd -g 1100 asmadmingroupadd -g 1200 dbagroupadd -g 1201 opergroupadd -g 1300 asmdbagroupadd -g 1301 asmoper创建用户Oracle 和griduseradd -g oinstall -G asmdba,asmadmin,asmoper,dba griduseradd -g oinstall -G asmdba,dba,oper oracle创建名文件安装路径，设置文件权限mkdir -p /u01/app/gridmkdir -p /u01/app/11.2.4/gridchown -R grid:oinstall /u01mkdir -p /u01/app/oraclechown oracle:oinstall /u01/app/oraclechmod -R 775 /u01修改用户密码passwd grid密码设置成grid123passwd oracle密码设置成oracle123设置用户环境变量root用户vim /root/.bash_profile添加下列信息export MANPATH=$MANPATH:/opt/VRTS/manPATH=/usr/sbin:/sbin:/usr/bin:/opt/VRTS/bin:$PATH; export PATH修改Oracle用户环境变量节点sfrac1vim /home/oracle/.bash_profile添加下列信息ORACLE_BASE=/u01/app/oracleORACLE_HOME=$ORACLE_BASE/product/11.2.4/dbhome_1ORACLE_SID=dbas1 export ORACLE_UNQNAME=racdbLD_LIBRARY_PATH=$ORACLE_HOME/libPATH=$PATH:$ORACLE_HOME/bin:$HOME/binexport ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATHexport LANG=C节点sfrac2vim /home/oracle/.bash_profile添加下列信息ORACLE_BASE=/u01/app/oracleORACLE_HOME=$ORACLE_BASE/product/11.2.4/dbhome_1ORACLE_SID=dbas2export ORACLE_UNQNAME=racdbLD_LIBRARY_PATH=$ORACLE_HOME/libPATH=$PATH:$ORACLE_HOME/bin:$HOME/binexport ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATHexport LANG=C修改grid用户环境变量vim /home/grid/.bash_profile添加下列信息export ORACLE_BASE=/u01/app/gridexport GRID_HOME=/u01/app/gridexport ORACLE_HOME=/u01/app/11.2.4/gridexport PATH=$PATH:$HOME/bin:$ORACLE_HOME/binumask 022export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/libLD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATHCLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATHexport LANG=C1.6 Storage Foundation For Oracle Rac安装和配置1.6.1 上传并解压SF上传安装包到/opt目录，并解压rootrac1 grid#tar -xzvf VRTS_SF_HA_Solutions_6.1_RHEL.tar.gz1.6.2 安装SF/opt/dvd1-redhatlinux/rhel6_x86_64/storage_foundation_for_oracle_rac/installsfrac输入y完全安装输入3输入主机名sfrac1 sfrac2输入用户密码，配置节点ssh通信确认信息并安装sf以下配置默认选NO1.6.3 配置SF运行/opt/VRTS/install/installsfrac61 -configure输入主机名配置组件，选1默认继续，回车输入唯一集群名racdb手动选择心跳，选1选择eth1和eth3，配置ID总揽，正确回车不配置VIP不用安全模式添加管理员用户信息，默认确认信息不配置SMTP不配置SNMP，确定停止VCS之后程序自动停止VCS和重启VCS，确认继续检查无报错验证配置rootsfrac1 # gabconfig -aGAB Port Memberships=Port a gen da9202 membership 01Port b gen da9201 membership 01Port d gen da9207 membership 01Port f gen da9213 membership 01Port h gen da9205 membership 01Port o gen da9209 membership 01Port u gen da9211 membership 01Port v gen da920b membership 01Port w gen da920e membership 01Port y gen da920c membership 01Port 功能说明：a GABb I/O 防护d ODM (Oracle Disk Manager)f CFS (Cluster File System)h VCS（Veritas Cluster Server：高可用性后台驻留程序）o VCSMM 驱动程序v CVM (Cluster Volume Manager)w vxconfigd（用于 CVM 的模块）1.6.4 配置IO防护使用vxdmpadm list dmpnode和fdisk -l 确定磁盘vxdmpadm list dmpnode 显示/dev/vx/dmp/下disk_n与/de/sdn之间的关系rootsfrac1 # vxdmpadm list dmpnodedmpdev = aluadisk0_15state = enabledenclosure = aluadisk0cab-sno = ALUAdiskasl = scsi3_jbodvid = UDsafepid = SDOP4100array-name = aluadiskarray-type = ALUAiopolicy = MinimumQavid = -lun-sno = 600D02310003634C6F9226D80DF20B97udid = UDsafe%5FSDOP4100%5FALUAdisk%5F600D02310003634C6F9226D80DF20B97dev-attr = -lun_type = -scsi3_vpd = -replicated = nonum_paths = 4#path = name state type transport ctlr hwpath aportID aportWWN attrpath = sdai enabled(a) primary FC c2 c2 - 21:00:00:d0:23:03:63:4c -path = sds enabled secondary FC c2 c2 - 21:00:00:d0:23:13:63:4c -path = sdi enabled(a) primary FC c1 c1 - 21:00:00:d0:23:03:63:4c -path = sdaa enabled secondary FC c1 c1 - 21:00:00:d0:23:13:63:4c -列出磁盘rootsfrac1 ntp# vxdisk -o alldgs listDEVICE TYPE DISK GROUP STATUSaluadisk0_0 auto:cdsdisk - (vxfendg) onlinealuadisk0_1 auto:cdsdisk aluadisk0_1 oradatadg online sharedaluadisk0_2 auto:cdsdisk - (vxfendg) onlinealuadisk0_3 auto:cdsdisk - (vxfendg) onlinealuadisk0_4 auto:cdsdisk aluadisk0_4 ocrvotedg online sharedaluadisk0_5 auto:cdsdisk aluadisk0_5 ftpdg online sharedaluadisk0_7 auto:cdsdisk - (vxfendgbak) onlinealuadisk0_8 auto:cdsdisk - (vxfendgbak) onlinealuadisk0_9 auto:cdsdisk - (vxfendgbak) onlinealuadisk0_10 auto:cdsdisk aluadisk0_10 dbbakdg online sharedaluadisk0_11 auto:cdsdisk aluadisk0_11 ftpbakdg online sharedaluadisk0_12 auto:none - - online invalidaluadisk0_13 auto:cdsdisk aluadisk0_13 oradatadgud online sharedaluadisk0_14 auto:cdsdisk aluadisk0_14 ftpdgud online sharedaluadisk0_15 auto:cdsdisk aluadisk0_15 ocrvotedgud online shareddisk_0 auto:LVM - - online invalid更具规划格式化磁盘vxdisksetup -i aluadisk0_0 format=cdsdiskvxdisksetup -i aluadisk0_1 format=cdsdiskvxdisksetup -i aluadisk0_2 format=cdsdiskvxdisksetup -i aluadisk0_3 format=cdsdiskvxdisksetup -i aluadisk0_4 format=cdsdiskvxdisksetup -i aluadisk0_5 format=cdsdiskvxdisksetup -i aluadisk0_7 format=cdsdiskvxdisksetup -i aluadisk0_8 format=cdsdiskvxdisksetup -i aluadisk0_9 format=cdsdiskvxdisksetup -i aluadisk0_10 format=cdsdiskvxdisksetup -i aluadisk0_11 format=cdsdiskvxdisksetup -i aluadisk0_13 format=cdsdiskvxdisksetup -i aluadisk0_14 format=cdsdiskvxdisksetup -i aluadisk0_15 format=cdsdisk通过脚本配置IO fencing/opt/VRTS/install/installsfrac61 -fencing制定sfrac1来进行配置确认信息继续选择2基于磁盘的io防护创建磁盘组vxfendg并选择磁盘继续，选择dmp确认信息，并重启配置代理，全部默认验证IO配置rootsfrac1 ntp# vxfenadm -dI/O Fencing Cluster Information:= Fencing Protocol Version: 201 Fencing Mode: SCSI3 Fencing SCSI3 Disk Policy: dmp Cluster Members: * 0 (sfrac1) 1 (sfrac2) RFSM State Information: node 0 in state 8 (running) node 1 in state 8 (running)1.7 ORACLE GI安装1.7.1 创建OCR，VOTEDISK和ORADATA磁盘注意：一下磁盘划分是根据某局某处安装编写的文档，其中磁盘组和逻辑卷的命名都统一，只有磁盘设备的选择需要根据各点的具体情况来划分。在sfrac1节点执行创建磁盘组rootsfrac1 # vxdg -s init ocrvotedg aluadisk0_4rootsfrac1 # vxdg -s init oradatadg aluadisk0_1rootsfrac1 # vxdg -s init ftpdg aluadisk0_5rootsfrac1 # vxdg -s init dbbakdg aluadisk0_10rootsfrac1 # vxdg -s init ftpbakdg aluadisk0_11rootsfrac1 # vxdg -s init oradatadgud aluadisk0_13rootsfrac1 # vxdg -s init ftpdgud aluadisk0_14rootsfrac1 # vxdg -s init ocrvotedgud aluadisk0_1分别创建逻辑卷rootsfrac1 # vxassist -g ocrvotedg make ocrvotevol maxsizerootsfrac1 # vxassist -g oradatadg make oradatavol maxsizerootsfrac1 # vxassist -g ftpdg make ftpvol maxsizerootsfrac1 # vxassist -g dbbakdg make dbbakvol maxsizerootsfrac1 # vxassist -g ftpbakdg make ftpbakvol maxsizerootsfrac1 # vxassist -g oradatadgud make oradatavoludmaxsizerootsfrac1 # vxassist -g ftpdgud make ftpvoludmaxsizerootsfrac1 # vxassist -g ocrvotedgud make ocrvotevoludmaxsize启动卷rootrac1 # vxvol -g ocrvotedg startallrootrac1 # vxvol -g oradatadg startallrootsfrac1 # vxvol -g ftpdg startallrootsfrac1 # vxvol -g dbbakdg startallrootsfrac1 # vxvol -g ftpbakdg startallrootsfrac1 # vxvol -g oradatadgud startallrootsfrac1 # vxvol -g ftpdgud startallrootsfrac1 # vxvol -g ocrvotedgud startall1.7.2 创建文件系统并挂载创建文件系统并挂载在同一挂载点在一个节点上执行mkfs -t vxfs /dev/vx/rdsk/ocrvotedg/ocrvotevolmkfs -t vxfs /dev/vx/rdsk/oradatadg/oradatavolmkfs -t vxfs /dev/vx/rdsk/ftpdg/ftpvolmkfs -t vxfs /dev/vx/rdsk/ftpbakdg/ftpbakvolmkfs -t vxfs /dev/vx/rdsk/dbbakdg/dbbakvolmkfs -t vxfs /dev/vx/rdsk/oradatadgud/oradatavoludmkfs -t vxfs /dev/vx/rdsk/ftpdgud/ftpvoludmkfs -t vxfs /dev/vx/rdsk/ocrvotedgud/ocrvotevolud在所有节点上执行mkdir /ocrmkdir /oradatamkdir /ftpmkdir /ftpbakmkdir /dbbakmount -t vxfs -o cluster /dev/vx/dsk/ocrvotedg/ocrvotevol /ocrmount -t vxfs -o cluster /dev/vx/dsk/oradatadg/oradatavol /oradatamount -t vxfs -o cluster /dev/vx/dsk/ftpdg/ftpvol /ftpmount -t vxfs -o cluster /dev/vx/dsk/ftpbakdg/ftpbakvol /ftpbak/mount -t vxfs -o cluster /dev/vx/dsk/dbbakdg/dbbakvol /dbbak/确认配置成功df -k1.7.3 将ocr和vote添加到VCS资源组修改VCS配置为读写权限haconf -makerw配置加入CVM资源组hares -add ocrvote_voldg_ocrvotedg CVMVolDg cvmhares -modify ocrvote_voldg_ocrvotedg Critical 0hares -modify ocrvote_voldg_ocrvotedg CVMDiskGroup ocrvotedghares -modify ocrvote_voldg_ocrvotedg CVMVolume -add ocrvotevolhares -modify ocrvote_voldg_ocrvotedg CVMActivation swVCS下配置文件系统hares -add ocrvote_mnt_ocrvotedg CFSMount cvmhares -modify ocrvote_mnt_ocrvotedg Critical 0hares -modify ocrvote_mnt_ocrvotedg MountPoint /ocrhares -modify ocrvote_mnt_ocrvotedg BlockDevice /dev/vx/dsk/ocrvotedg/ocrvotevol链接资源关系hares -link ocrvote_voldg_ocrvotedg cvm_clushares -link ocrvote_mnt_ocrvotedg ocrvote_voldg_ocrvotedghares -link ocrvote_mnt_ocrvotedg vxfsckd开启资源hares -modify ocrvote_voldg_ocrvotedg Enabled 1hares -modify ocrvote_mnt_ocrvotedg Enabled 1修改VCS配置为只读haconf -dump -makero将资源启动hares -online ocrvote_voldg_ocrvotedg -sys sfrac1hares -online ocrvote_voldg_ocrvotedg -sys sfrac2hares -online ocrvote_mnt_ocrvotedg -sys sfrac1hares -online ocrvote_mnt_ocr