Centos5.5+heartbeat+drbd+oracle10g双机配置实例.doc

Centos5.5+heartbeat+drbd+oracle10g双机配置实例一、安装环境说明：操作系统：centos5.5x86主要软件及版本：heartbeat2.0.8 drbd-8.4.0依赖软件及版本：libnet 1.1.2.1（heartbeat所需） 数据库版本：oracle 10201 32位安装前规划：网卡：两台机器配双网卡，一个用于连接公用网络，一个用于做心跳测试。主机名：分别为drbd1， drbd2，虚拟ip主机名drbdIP：主drbd1：eth0 10.10.10.11 心跳ip eth1 135.255.10.211 连接公网。从drbd2：eth0 10.10.10.12 心跳ip eth1 135.255.10.212 连接公网。两台机器的eth0用交叉线相连，eth1为局域网络，共用一个网关：130.255.10.1虚拟ip：eth1:0 135.255.10.200 虚拟ip由heartbeat自动管理，不需要预先设置。磁盘：两台机器分别加上一个大小20G的硬盘。用来做drbd网络共享存储，并在上面安装oracle。二、安装前配置ip地址的配置：eth0分别配置心跳ip，eth1分别配上上述ip地址。两台机器上主机名及网关的配置，这里以主机器为例：rootdrbd1 # cat /etc/sysconfig/networkNETWORKING=yesNETWORKING_IPV6=yesHOSTNAME=drbd1 #主机名 另一台机器为drbd2GATEWAY=135.255.10.1 #网关在两台机器上，都配置hosts文件，新增三个ip到主机名的解析rootdrbd1 # cat /etc/hosts# Do not remove the following line, or various programs# that require network functionality will fail.127.0.0.1 drbd1 localhost.localdomain localhost:1 localhost6.localdomain6 localhost6135.255.10.211 drbd1135.255.10.212 drbd2135.255.10.200 drbd主机名配置完成后，重启电脑，配置生效。三、安装heartbeat两台机器上做一样的安装和配置添加用户和用户组rootdrbd1 # groupadd haclientrootdrbd1 # useradd -g haclient hacluster首先安装libnetrootdrbd1 # cd /usr/local/src/rootdrbd1 src# tar zxvf libnet-1.1.2.1.tar.gzrootdrbd1 src# cd libnet/rootdrbd1 libnet# ./configurerootdrbd1 libnet# makerootdrbd1 libnet# make install安装heartbeatrootdrbd1 src# tar zxvf Heartbeat-2-1-2d298bca0d0a.tar.gzrootdrbd1 src# cd Heartbeat-2-1-2d298bca0d0a/rootdrbd1 Heartbeat-2-1-2d298bca0d0a# ./ConfigureMe configurerootdrbd1 Heartbeat-2-1-2d298bca0d0a# make注意，这里make到一半，可能会报一个类似hbaping HBcomm等字样的错误，解决方法：rootdrbd1 Heartbeat-2-1-2d298bca0d0a# cd lib/plugins/HBcomm/rootdrbd1 HBcomm# mv hbaping.loT hbaping.lo回到heartbeat文件根目录下重新再执行make即可继续编译。注意这里的hbaping.loT在第一次make之前是不存在的，必须要先make编译，生成了hbaping.loT，再在报错的时候文件名修改为hbaping.lo，再重新make一次即可。最后执行make install安装rootdrbd1 Heartbeat-2-1-2d298bca0d0a# make install复制三个配置文件到/etc/ha.d/目录下rootdrbd1 Heartbeat-2-1-2d298bca0d0a# cp doc/ha.cf doc/haresources doc/authkeys /etc/ha.d/到此，两台机器上的heartbeat就安装完成，先不做配置，继续安装drbd四、安装配置drbd先将两台机器的新硬盘进行分区，使用fdisk，给第二块硬盘，即/dev/sdb分区，这里都只划一个主分区/dev/sdb1，完成后先不进行格式化，稍后会用到。两台机器都安装配置drbd，配置完全相同：rootdrbd1 src# tar zxvf drbd-8.4.0.tar.gzrootdrbd1 src# cd drbd-8.4.0/rootdrbd1 drbd-8.4.0# ./configurerootdrbd1 drbd-8.4.0# make rpmrootdrbd1 drbd-8.4.0# make km-rpmrootdrbd1 drbd-8.4.0# cd /usr/src/redhat/RPMS/i386/rootdrbd1 i386# lsdrbd-8.4.0-1.i386.rpm drbd-heartbeat-8.4.0-1.i386.rpm drbd-pacemaker-8.4.0-1.i386.rpm drbd-xen-8.4.0-1.i386.rpmdrbd-bash-completion-8.4.0-1.i386.rpm drbd-km-2.6.18_194.el5-8.4.0-1.i386.rpm drbd-udev-8.4.0-1.i386.rpmdrbd-debuginfo-8.4.0-1.i386.rpm drbd-km-debuginfo-8.4.0-1.i386.rpm drbd-utils-8.4.0-1.i386.rpmrootdrbd1 i386# rpm -ivh drbd*rootdrbd1 i386# modprobe drbd #加载drbd到内核rootdrbd1 i386# lsmod |grep drbd #查看是否加载成功drbd 250852 0开始配置drbd主配置文件为/etc/drbd.confrootdrbd1 # cat /etc/drbd.conf# You can find an example in /usr/share/doc/drbd./drbd.conf.exampleinclude drbd.d/global_common.conf;include drbd.d/*.res;可以看到，主配置文件是通过包含/etc/drbd.d/global_common.conf全局通用配置文件和以.res结尾的这些/etc/drbd.d/*.res资源文件来配置drbd的。修改global_common.conf文件如下：global usage-count yes;common handlers pri-on-incon-degr /usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b /proc/sysrq-trigger ; reboot -f; pri-lost-after-sb /usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b /proc/sysrq-trigger ; reboot -f; local-io-error /usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o /proc/sysrq-trigger ; halt -f; startup degr-wfc-timeout 120; options # cpu-mask on-no-data-accessible disk on-io-error detach; net protocol C; syncer rate 100M; 资源文件配置如下：rootdrbd1 # cat /etc/drbd.d/r0.resresource r0 #资源名称为r0 on drbd1 #drbd1上面的配置 device /dev/drbd1;#drbd1上的drbd设备 disk /dev/sdb1;#drbd1上的物理磁盘分区 address 10.10.10.11:7789;#本机ip地址，这里选择心跳ip meta-disk internal;#表示在同一局域网内 on drbd2 #drbd2机器上的配置 device /dev/drbd1; disk /dev/sdb1; address 10.10.10.12:7789; meta-disk internal; 两台机器上都执行如下命令创建r0资源数据块rootdrbd1 # drbdadm create-md r0md_offset 21467942912al_offset 21467910144bm_offset 21467254784Found ext3 filesystem 20964116 kB data area apparently used 20964116 kB left usable by current configurationEven though it looks like this would place the new meta data intounused space, you still need to confirm, as this is only a guess.Do you want to proceed?need to type yes to confirm yesYou want me to create a v08 style flexible-size internal meta data block.There appears to be a v08 flexible-size internal meta data blockalready in place on /dev/sdb1 at byte offset 21467942912Do you really want to overwrite the existing v08 meta-data?need to type yes to confirm yesWriting meta data.initializing activity logNOT initializing bitmapNew drbd meta data block successfully created.两次输入yes，创建成功。再在两台机器上分别启动drbdrootdrbd1 # /etc/init.d/drbd start查看drbd运行状态：rootdrbd1 # cat /proc/drbdversion: 8.4.0 (api:1/proto:86-100)GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by rootdrbd1, 2011-12-27 15:00:40 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:20964116发现在两台机器上，drbd都为Secondary模式，因此需要指定哪个为主，哪个为从设备只在drbd1机器上执行如下命令：rootdrbd1 # drbdsetup /dev/drbd1 primary force查看drbd运行状态，发现drbd正在同步数据rootdrbd1 # cat /proc/drbdversion: 8.4.0 (api:1/proto:86-100)GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by rootdrbd1, 2011-12-27 15:00:40 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r- ns:1149952 nr:0 dw:0 dr:1158144 al:0 bm:69 lo:3 pe:1 ua:8 ap:0 ep:1 wo:b oos:19815188 . synced: 5.6% (19348/20472)M finish: 0:04:35 speed: 71,808 (71,808) K/sec这里需要等待一点时间，让数据同步完成，然后执行如下命令来使drbd1为drbd的主设备，只在drbd1上执行：rootdrbd1 # drbdadm primary r0再看drbd的运行状态rootdrbd1 # cat /proc/drbdversion: 8.4.0 (api:1/proto:86-100)GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by rootdrbd1, 2011-12-27 15:00:40 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-ns:20964116 nr:0 dw:0 dr:20964116 al:0 bm:1280 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0可以看到drbd1现在为primary了，drbd2上正好相反：rootdrbd2 # cat /proc/drbdversion: 8.4.0 (api:1/proto:86-100)GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by rootdrbd2, 2011-12-27 15:06:19 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-ns:0 nr:20964116 dw:20964116 dr:0 al:0 bm:1280 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0drbd1上查看角色rootdrbd1 # drbdadm role r0Primary/Secondarydrbd2上查看角色rootdrbd2 # drbdadm role r0Secondary/Primarydrbd设备，同一时间，只能在primary机器上被使用，因此这里只在drbd1上操作，对/dev/drbd1进行格式化，并挂载rootdrbd1 # mkfs -t ext3 /dev/drbd1rootdrbd1 # mkdir /oradate #在drbd2上面也要新建这个目录rootdrbd1 # mount /dev/drbd1 /oradate/到此，drbd配置完成，下面需要在drbd设备的挂载点/oradate安装oracle五、安装oracle10g在两台机器上，做同样的oracle安装前的配置，下面以drbd1为例：oracle安装详细参考官方文档，下面作简要的配置查看是否安装libXp，如果没有安装，则用yum进行安装：rootdrbd1 # rpm aq | grep libXprootdrbd1 # yum install libXp创建用户和用户组：rootdrbd1 # groupadd oinstallrootdrbd1 # groupadd dbarootdrbd1 # useradd -g oinstall -G dba oraclerootdrbd1 # passwd oracle在/etc/sysctl.conf里面修改添加如下参数kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000 net.core.rmem_default = 1048576 net.core.rmem_max = 1048576 net.core.wmem_default = 262144 net.core.wmem_max = 262144保存后执行如下命令应用参数rootdrbd1 # sysctl -p为oracle用户设置限制，加入/etc/security/limits.conf文件，如下：oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536在/etc/pam.d/login文件加入如下内容：session required /lib/security/pam_limits.so session required pam_limits.so编辑/etc/profile 末尾加入如下脚本：if $USER = oracle ; then if $SHELL = /bin/ksh ; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fifi给oracle BASE目录赋权限rootdrbd1 # chown R oracle:oinstall /oradaterootdrbd1 # chmod R 755 /oradate在oracle家目录下oracle/.bash_profile加入如下环境变量：ORACLE_BASE=/oradateORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1ORACLE_SID=orclPATH=$PATH:$HOME/bin:$ORACLE_HOME/binLD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib:/lib:/usr/libexport ORACLE_BASE ORACLE_HOME ORACLE_SID PATH LD_LIBRARY_PATH修改系统版本文件/etc/redhat-release把如下内容 CentOS release 5.5 (Final)改成Redhat 4保存解压缩oracle安装包，注销root，用oracle登录，并启动图形界面oracledrbd1 $ startx在图形界面下，开启一个终端，临时修改系统字符变量为英文oracledrbd1 $ export LANG=en_US进入oracle安装包目录，开始安装oracleoracledrbd1 $ cd /opt/database/oracledrbd1 database$ ./runInstaller详细图形安装过程参考官方文档安装完成后，对oracle监听进行配置，修改后如下oracledrbd1 database$ cd $ORACLE_HOME/network/adminoracledrbd1 database$ cat listener.oraSID_LIST_LISTENER = (SID_LIST = (SID_DESC = (SID_NAME = PLSExtProc) (ORACLE_HOME = /oradate/product/10.2.0/db_1) (PROGRAM = extproc) )(SID_DESC = (GLOBAL_DBNAME = orcl) (ORACLE_HOME = /oradate/product/10.2.0/db_1) (SID_NAME = orcl) ) )LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1) (ADDRESS = (PROTOCOL = TCP)(HOST = drbd)(PORT = 1521) ) )LISTENER的HOST部分，填上虚拟ip或者虚拟ip的主机名，这里为主机名drbd关闭监听，关闭数据库，下面开始配置heartbeat六、配置heartbeat在drbd1上进行配置rootdrbd1 # cd /etc/ha.d/1、双机互联验证文件rootdrbd1 ha.d# grep -v # authkeys | grep -v $auth 11 crc注意设置authkeys文件的权限为600rootdrbd1 ha.d# chmod 600 authkeys2、主配置文件ha.cf，去掉注释和空行后，配置如下rootdrbd1 ha.d# grep -v # ha.cf | grep -v $logfile /var/log/ha-loglogfacility local0keepalive 2deadtime 30warntime 10initdead 120udpport 694baud 19200ucast eth0 10.10.10.12 #此处选择单播方式进行通信，ip地址为对方节点的心跳ipauto_failback onnode drbd1node drbd2ping 135.255.10.1 #一个能正常ping通的ip，这里选择固定的网关respawn hacluster /usr/lib/heartbeat/ipfailapiauth ipfail gid=haclient uid=hacluster3、资源配置文件，只有一行rootdrbd1 ha.d# grep -v # haresources | grep -v $drbd1 IPaddr:135.255.10.200/24/eth1 drbddisk:r0 Filesystem:/dev/drbd1:/oradate:ext3 notechange.sh说明：drbd1为主节点的主机名，从节点也同样的配置，IPaddr为heartbeat对虚拟ip的设置，drbddisk设置drbd资源的角色，Filesystem挂载drbd磁盘，notechange.sh为共享磁盘挂载卸载，oracle启动关闭脚本，脚本存放在/etc/ha.d /resource.d下，脚本内容如下：rootdrbd1 ha.d# cat resource.d/notechange.sh#!/bin/sh#Its used to startup/shutdown oracle databaseexport ORACLE_BASE=/oradateexport ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1export ORACLE_SID=orclexport PATH=$PATH:$HOME/bin:$ORACLE_HOME/binexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib:/lib:/usr/libMOUNTDEV=/dev/drbd1MOUNTPOINT=/oradatecase $1 instart)/bin/mount $MOUNTDEV $MOUNTPOINTsu - oracle -c dbstart;stop)su oracle c dbshutfuser -k -m -v $MOUNTPOINT/bin/umount $MOUNTPOINT;*)echo Usage: $0 start|stop;esac其中，需要说明的是dbstart和dbshut两个脚本，这两个脚本为安装数据库后，自动生成的脚本，用来启动，关闭监听和数据库，其中dbstart部分地方需要做修改，才能正常启动。修改如下：ORACLE_HOME_LISTNER=$ORACLE_HOME脚本通过ORACLE_HOME_LISTNER的值来定位到监听程序的可执行程序，从而启动监听，由于默认脚本设置的值和自己安装的目录路径不一样，因此需要修改成当前oracle的安装家目录。另一个地方需要修改的是/etc/oratab文件，dbstart脚本通过读取oratab文件来决定是否启动数据库实例，启动哪个数据库实例。修改如下：orcl:/oradate/product/10.2.0/db_1:Y将最后面的N改成Y，则为要启动orcl实例，路径为/oradate/product/10.2.0/db_1如果其他还需要修改的，仔细阅读脚本，修改成当前oracle安装环境的变量即可。在主节点上配置完heartbeat后，再把这些配置文件全部复制到从节点，因为从节点的配置文件要和主节点完全相同：rootdrbd2 # scp -r drbd1:/etc/ha.d/* /etc/ha.d/注意ha.cf文件里面单播通讯的地方，ip地址要填对方节点的心跳ip，除了这里需要修改外，其他地方完全一样！复制主节点/etc/oratab文件到从节点rootdrbd2 # scp drbd1:/etc/ora* /etc/这里将主节点上安装数据库时生成的两个文件都拷贝过来，包括修改后的oratab文件。七、启动heartbeat查看drbd状态：rootdrbd1 # cat /proc/drbdversion: 8.4.0 (api:1/proto:86-100)GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by rootdrbd1, 2011-12-27 15:00:40 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-ns:20964644 nr:0 dw:528 dr:20964373 al:7 bm:1280 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0此时，drbd已经在正常运行，可以启动heartbeat了。两台机器上，分别启动heartbeatrootdrbd1 # /etc/init.d/heartbeat startStarting High-Availability services:2012/01/07_20:32:25 INFO: Resource is stopped 确定rootdrbd1 #稍等片刻，查看drbd是否挂载rootdrbd1 # ls /oradate/admin flash_recovery_area lost+found oradata oraInventory product可以看到，oradate目录下已经有了oracle的安装文件了rootdrbd1 # mount/dev/sda2 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sda3 on /usr type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw).host:/ on /mnt/hgfs type vmhgfs (rw,ttl=1)none on /proc/fs/vmblock/mountPoint type vmblock (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)/dev/drbd1 on /oradate type ext3 (rw)最后一行，表明已经挂载，再来看oracle的监听是否开启，切换到oracle用户，查看监听的状态：rootdrbd1 # su - oracleoracledrbd1 $ lsnrctl statusLSNRCTL for Linux: Version 10.2.0.1.0 - Production on 07-JAN-2012 20:35:06Copyright (c) 1991, 2005, Oracle. All rights reserved.Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)STATUS of the LISTENER-Alias LISTENERVersion TNSLSNR for Linux: Version 10.2.0.1.0 - ProductionStart Date 07-JAN-2012 20:33:00Uptime 0 days 0 hr. 2 min. 5 secTrace Level offSecurity ON: Local OS AuthenticationSNMP OFFListener Parameter File /oradate/product/10.2.0/db_1/network/admin/listener.oraListener Log File /oradate/product/10.2.0/db_1/network/log/listener.logListening Endpoints Summary. (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=135.255.10.200)(PORT=1521)Services Summary.Service PLSExtProc has 1 instance(s). Instance PLSExtProc, status UNKNOWN, has 1 handler(s) for this service.Service orcl has 1 instance(s). Instance orcl, status UNKNOWN, has 1 handler(s) for this service.The command completed successfully监听正常，登录数据库，查看数据库的运行状态：oracledrbd1 $ sqlplus / as sysdbaSQL*Plus: Release 10.2.0.1.0 - Production on Sat Jan 7 20:35:48 2012Copyright (c) 1982, 2005, Oracle. All rights reserved.Connected to:Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - ProductionWith the Partitioning, OLAP and Data Mining optionsSQL select database_status from v$instance;DATABASE_STATUS-ACTIVESQL show user;USER is SYSSQL !hostnamedrbd1数据库开启并且活跃，用户为sys，当前服务器主机名为drbd1，即主节点。八、测试heartbeat切换是否正常关闭主节点drbd1，稍后待heartbeat切换完成后，到节点2上查看drbd运行状态和数据库的运行状态rootdrbd2 # cat /proc/drbdversion: 8.4.0 (api:1/proto:86-100)GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by rootdrbd2, 2011-12-27 15:06:19 1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-ns:0 nr:20984860 dw:20985092 dr:7561 al:10 bm:1280 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:204可以看到，节点2的drbd当前为primary状态，从节点为unknownrootdrbd2 # ls /oradate/admin flash_recovery_area lost+found oradata oraInventory product目录下已经有文件了rootdrbd2 # mount/dev/sda2 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sda3 on /usr type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw).host:/ on /mnt/hgfs type vmhgfs (rw,ttl=1)none on /proc/fs/vmblock/mountPoint type vmblock (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)/dev/drbd1 on /oradate type ext3 (rw)最后一行表明已经drbd设备已经挂载rootdrbd2 # su - oracleoracledrbd2 $ lsnrctl statusLSNRCTL for Linux: Version 10.2.0.1.0 - Production on 07-JAN-2012 20:42:33Copyright (c) 1991, 2005, Oracle. All rights r