RAC平台搭建设计方案.docx

RAC平台搭建设计方案第1章RAC系统结构设计1.1Oracle RAC架构图图1-1Oracle RAC架构图1.2Oracle RAC硬件信息Oracle RAC环境由两台服务器搭建，每台主机至少配置了两块网卡，其中一块网卡用于向外提供服务，另一块网卡用于两台节点之间的心跳。服务器操作系统数据库版本内存CPU网卡共享存储DB1Redhalt 6.6Oracle.032G4P*12C2块hbaVotdisk 5G2块ethDatadisk1 200GDB2Redhalt 6.6Oracle.032G4P*12C2块hbaDatadisk1 200G2块ethFlashdisk 100G1.3Oracle RAC各节点信息节点名IP地址类型主机名主机别名网卡接口ODB01IP1PublicIPODB01ODB01.eth0IP1:1VirtualIPODB01-vipODB01-eth0:1Priv IP1PrivateIPODB01-privODB01-eth1ODB02IP2PublicIPODB02ODB02.eth0IP2:2VirtualIPODB02-vipODB02-eth0:1Priv IP2PrivateIPODB02-privODB02-eth1SCAN-IP10.0.144.SCANIPeth0第2章RAC安装环境配置2.1Oracle RAC软件组件信息软件组织名称OS用户主组辅组用户目录oracl基目录gridgrid oinstall（1000）dba(1200)asmadmin（1100）asmdba（1300）asmoper（1301）/u01/app/11.2.0/grid/u01/app/gridoracle racoracleoinstall（1000）dba（1200）asmdba（1300）/u01/app/oracle/product/11.2.0/db_1/u01/app/oracle2.2Oracle RAC用户组信息组名属组用户Oracle权限Oracle组名描述oinstallOraclegridOracle清单和软件所有者dbaOraclegridsysdba数据库管理员asmadmingridsystemosasm自劢存储管理组asmdbaOraclegridAsm sysdbaOsdba for asmASM数据库管理员组asmopergridAsm sysoperOsoper for asmASM操作员组2.3Oracle RAC软件清单1) Red Hat Enterprise Linux AS release 6.63) p13390677_112040_Linux-x86-64_1of7.zip(database)4) p13390677_112040_Linux-x86-64_2of7.zip(database)5) p13390677_112040_Linux-x86-64_3of7.zip(grid)6) oracleasm-2.6.18-371.el5-2.0.5-1.el5.x86_64.rpm7)oracleasm-2.6.18-371.el5debug-2.0.5-1.el5.x86_64.rpm8)oracleasm-2.6.18-371.el5xen-2.0.5-1.el5.x86_64.rpm9)oracleasmlib-2.0.4-1.el5.x86_64.rpm10)oracleasm-support-2.1.8-1.el5.x86_64.rpm2.4网络规划两个节点的网络配置，包括网卡配置和/etc/hosts表的写法。ODB01服务器 1 (ODB01) 设备IP地址子网网关备注Eth0IP 1xxx.xxx.xxx.xxxxxx.xxx.xxx.xxxPublic IPEth1Priv IP1xxx.xxx.xxx.xxxxxx.xxx.xxx.xxxPRIVATE IPEth0:1IP1:1xxx.xxx.xxx.xxxxxx.xxx.xxx.xxxVIP/etc/hosts#eth0-Public IP#IP1ODB01ODB01.IP2ODB02ODB02.# eth1-Private IP #Priv-ip1ODB01-priv ODB01-Priv-ip2ODB02-priv ODB02-# VIP#IP1:1ODB01-vip ODB01-IP2:2ODB02-vip ODB02-# scan-cluster #Scan IP scan-cluster ODB02服务器 1 (ODB02) 设备IP地址子网网关备注Eth0IP2xxx.xxx.xxx.xxxxxx.xxx.xxx.xxxPublic IPEth1Priv-IP2xxx.xxx.xxx.xxxxxx.xxx.xxx.xxxPRIVATE IPEth0:1IP2:2xxx.xxx.xxx.xxxxxx.xxx.xxx.xxxVIP/etc/hosts#eth0-Public IP#42ODB01ODB01.43ODB02ODB02.# eth1-Private IP #ODB01-priv ODB01-ODB02-priv ODB02-# VIP#44ODB01-vip ODB01-45ODB02-vip ODB02-# scan-cluster #46 scan-cluster 2.5配置DNS域名解析服务 安裝DNS软件包rootODB01 dns# rpm -ivh bind-9.3.6-20.P1.el5.x86_64.rpmwarning: bind-9.3.6-20.P1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 37017186Preparing. # 100%package bind-9.3.6-20.P1.el5.x86_64 is already installedrootODB01 dns# rpm -ivh bind-chroot-9.3.6-20.P1.el5.x86_64.rpmwarning: bind-chroot-9.3.6-20.P1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 37017186Preparing. # 100%package bind-chroot-9.3.6-20.P1.el5.x86_64 is already installedrootODB01 dns# rpm -ivh bind-devel-9.3.6-20.P1.el5.x86_64.rpmwarning: bind-devel-9.3.6-20.P1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 37017186Preparing. # 100% 1:bind-devel # 100%rootODB01 dns# rpm -ivh caching-nameserver-9.3.6-20.P1.el5.x86_64.rpmwarning: caching-nameserver-9.3.6-20.P1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 37017186Preparing. # 100% 1:caching-nameserver # 100%配置/var/named/chroot/etc/named.conf文件先拷贝生成该文件：rootODB02 dns# cd /var/named/chroot/etcrootODB02 etc# cp p named.caching-nameserver.conf named.conf说明：为简单起见，将该/var/named/chroot/etc/named.conf文件中的、localhost全部修改成any，且修改时，需要注意左右两边留空格。修改部分在下述配置文件中以红色加粗标示。rootODB01 etc# cat named.conf/ named.caching-nameserver.conf/ Provided by Red Hat caching-nameserver package to configure the/ ISC BIND named(8) DNS server as a caching only nameserver / (as a localhost DNS resolver only). / See /usr/share/doc/bind*/sample/ for example named configuration files./ DO NOT EDIT THIS FILE - use system-config-bind or an editor/ to create named.conf - edits to this file will be lost on / caching-nameserver package upgrade./options listen-on port 53 any; ;listen-on-v6 port 53 :1; ;directory /var/named;dump-file /var/named/data/cache_dump.db; statistics-file /var/named/data/named_stats.txt; memstatistics-file /var/named/data/named_mem_stats.txt;/ Those options should be used carefully because they disable port/ randomization/ query-source port 53;/ query-source-v6 port 53;allow-query any; ;allow-query-cache any; ;logging channel default_debug file data/named.run; severity dynamic; ;view localhost_resolver match-clients any; ;match-destinations any; ;recursion yes;include /etc/named.rfc1912.zones;配置Zone文件修改/var/named/chroot/etc/named.rfc1912.zones文件，配置正反向解釋zone，配置Zone的目的是为了解析SCANIP，在该文件末尾添加以下反向Zone：#反向解析#zone 144.0.10. IN type master;file 144.0.10.;allow-update none; ;配置Zone的目的是为了解析域名，在该文件末尾添加以下正向Zone：#正向解析#zone IN type master;file scan-cluster.zone;allow-update none; ;配置正、反向解析数据库文件在/var/named/chroot/var/named路径下进行配置，首先，生成正、反向解析数据库文件。利用/var/named/chroot/var/named/localhost.zone复制生成scan-cluster.zone的正向解析的数据库文件，利用/var/named/chroot/var/named/named.local复制生成名为144.0.10.的反向解析的数据库文件。然后，定义正、反向解析数据库文件。rootODB01 etc# cd /var/named/chroot/var/named/rootODB01 named# cp p localhost.zone scan-cluster.zonerootODB01 named# cp p named.local 147.0.10.说明：生成的正、反向解析数据库文件名一定要同上述步骤中定义的正、反向Zone文件一致，否则DNS出错！正向解析数据库文件scan-cluster.zone文件，如下所示：$TTL 86400 IN SOA ( 42 ; serial (d. adams) 3H ; refresh 15M ; retry 1W ; expiry 1D ) ; minimum IN NS IN A IN AAAA :1scan-cluster IN A 46反向解析数据库144.0.10.文件如下所示：$TTL86400 IN SOA . . ( 1997022700 ; Serial 28800 ; Refresh 14400 ; Retry 3600000 ; Expire 86400 ) ; Minimum IN NS .1 IN PTR localhost.146 IN PTR .重启DNS服务并且设置系统启动自动加载启动DNS服务：rootODB01 #service named restartStopping named: OK Starting named: OK 设置系统启动自动加载DNS服务:rootODB01 #chkconfig -level 2345 named on配置/etc/resolv.conf分别在RAC所有节点/etc/resolv.conf配置文件中添加下述配置信息：nameserver42nameserver 43Nslookup测试ODB01：rootODB01 # nslookup Server:ScanIPAddress:ScanIP #53Name:Address: xxx.xxx.xxx.xxxrootODB01 # nslookup 46Server:43Address:43#530.name = .ODB02：rootODB02# nslookup Server:43Address:43#53Name:Address: 19rootODB02 # nslookup 46Server:43Address:43#530.name = .说明：为了实现DNS服务器冗余，按照上述操作在ODB02节点安装DNS服务。业务数据库配置参数ORACLE_BASE/u01/app/gridCRS_HOME/u01/app/11.2.0/gridORACLE_HOME/u01/app/oracle/product/11.2.0/db_1DB_HOMEODBORACLE_SIDOdb1,odb2TNS_ADMIN$ORACLE_HOME/network/adminOracle管理账户的口令Sys/orasys或者system/orasys数据库存放位置asm是否使用归档方式运行数据库Enable备份方式说明RMAN2.6系统安装规划第3章安装RAC3.1安装Oracle RAC需求的系统软件包安装Oracle RAC需要系统安装额外的软件包，可以下面的命令安装以下软件包。yum install -y binutils compat-libstdc+ elfutils-libelf elfutils-elf-devel elfutils-libelf-devel-static gcc gcc-c+ glibc glibc-common glibc-devel glibc-headers kernel-headers ksh libaio libaio-devel libgcc libgomp libstdc+ libstdc+-devel make sysstat unixODBC unixODBC-devel3.2设置主机名在两个节点配置/ec/sysconfig/network，修改主机名。rootODB01 # vi /etc/sysconfig/networkNETWORKING=yesNETWORKING_IPV6=noHOSTNAME=ODB01 #此处为主机名GATEWAY=333.3创建Oracle RAC相关的用户和组创建RAC相关用户和组，在所有节点执行。rootODB01 #groupadd -g 1000 oinstallrootODB01 #groupadd -g 1100 asmadminrootODB01 #groupadd -g 1200 dbarootODB01 #groupadd -g 1201 operrootODB01 #groupadd -g 1300 asmdbarootODB01 #groupadd -g 1301 asmoperrootODB01 #useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid -s /bin/bash gridrootODB01 #useradd -u 1101 -g oinstall -G dba,asmdba -d /home/oracle -s /bin/bash oracle3.4修改用户密码修改新创建的用户oracle和grid用户的密码，在所有节点执行。rootODB01 # passwd oraclerootODB01 # passwd grid 3.5修改gird用户的环境变量修改grid用户环境变量，在所有节点执行。切换到 grid 用户，准备安装 grid Infrastructure。首先需要修改所有节点 grid 用户的环境变量.bash_profile。rootODB01 init.d# su gridgridODB01 $ cd /home/grid/ 以下grid的.bash_profile,仅供参考，还有一些参数可以自行加入。export PS1=/bin/hostname -s(grid)- export TMP=/tmpexport TMPDIR=$TMPexport ORACLE_SID=+ASM1export ORACLE_BASE=/u01/app/gridexport ORACLE_HOME=/u01/app/11.2.0/gridexport ORACLE_TERM=xtermTHREADS_FLAG=native; export THREADS_FLAG export NLS_DATE_FORMAT=yyyy-mm-dd hh24:mi:ssexport TNS_ADMIN=$ORACLE_HOME/network/adminexport PATH=/usr/sbin:$PATHexport PATH=$ORACLE_HOME/bin:$PATHexport LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/libexportCLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlibexport EDITOR=viexport LANG=en_USexport NLS_LANG=american_america.AL32UTF8umask 022注意：使配置文件生效，同样在其他节点的Grid用户.bash_profile 中加入，并注意修改 ORACLE_SID=+ASM2 3.6修改oracle用户环境变量Vi /home/oracle/.bash_profileexport PATHexport PS1=/bin/hostname -s(oracle)- export TMP=/tmpexport TMPDIR=$TMPexport ORACLE_HOSTNAME=ODB01.export ORACLE_SID=odb01export ORACLE_BASE=/u01/app/oracleexport ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1export ORACLE_UNQNAME=odbexport TNS_ADMIN=$ORACLE_HOME/network/adminexport ORACLE_TERM=xtermexport PATH=/usr/sbin:$PATHexport PATH=$ORACLE_HOME/bin:$PATHexport LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/libexport CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlibexport EDITOR=viexport NLS_LANG=AMERICAN_AMERICA.ZHS16GBKexport NLS_DATE_FORMAT=yyyy-mm-dd hh24:mi:ssumask 022注意：使配置文件生效, 同样在其他节点的Oracle用户.bash_profile 中加入，并修改 ORACLE_SID=odb023.7创建软件目录创建Oracle RAC所使用的目彔，并修改相应的权限，在所有节点执行；rootODB01 #mkdir -p /u01/app/grid rootODB01#mkdir -p /u01/app/11.2.0/grid rootODB01 #mkdir -p /u01/app/oracle rootODB01 #chown -R oracle:oinstall /u01rootODB01 #chown -R grid:oinstall /u01/app/grid rootODB01 #chown -R grid:oinstall /u01/app/11.2.0rootODB01 #chmod -R 775 /u013.8修改系统参数修改/etc/sysctl.conf文件kernel.shmall = 4294967296fs.aio-max-nr = 1048576fs.file-max = 6815744kernel.shmall = 2097152kernel.shmmax = 1054472192(100G)kernel.shmmni = 4096kernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048586net.ipv4.tcp_wmem = 262144 262144 262144net.ipv4.tcp_rmem = 4194304 4194304 4194304注意：重启内核 sysctl p修改/etc/security/limits.conforacle soft nproc 2047oracle hard nproc 16384oracle soft nofile 1024oracle hard nofile 65536grid soft nproc 2047grid hard nproc 16384grid soft nofile 1024grid hard nofile 65536修改/etc/pam.d/loginsession required /lib/security/pam_limits.sosession required pam_limits.so修改/etc/profileif $USER = oracle | $USER = grid ; thenif $SHELL = /bin/ksh ; thenulimit -p 16384ulimit -n 65536elseulimit -u 16384 -n 65536fifi修改/etc/csh.loginif ( $USER = oracle | $USER = grid ) then limit maxproc 16384 limit descriptors 65536endif EOFCSH3.9配置用户等效性使用grid用户，配置grid用户等效性，在所有节点执行。配置grid用户ssh等效性（在所有节点配置）rootODB01 #su gridODB01(grid)-mkdir /.sshODB01(grid)-chmod 755 /.sshODB01(grid)-ssh-keygen -t dsa#生成dsa公钥 (直接回車不需要输入密码)ODB01(grid)-ssh-keygen -t rsa #生成dsa公钥(直接回車不需要输入密码)ODB01：ODB01(grid)-cat /.ssh/id_rsa.pub /.ssh/authorized_keysODB01(grid)-cat /.ssh/id_dsa.pub /.ssh/authorized_keys scp /.ssh/authorized_keys 43:/home/grid/.ssh/ #(输入ODB02密码，即可复制公钥到ODB02节点上)ODB02：ODB01(grid)-cat /.ssh/id_rsa.pub /.ssh/authorized_keysODB01(grid)-cat /.ssh/id_dsa.pub /.ssh/authorized_keys ODB01(grid)-scp /.ssh/authorized_keys 42:/home/grid/.ssh/ #(输入ODB01密码，即可复制到ODB02节点上)使用grid用户，配置grid用户等效性，在所有节点执行。配置oracle用户ssh等效性（在所有节点配置）rootODB01 #su gridODB01(oracle)-mkdir /.sshODB01(oracle)-chmod 755 /.sshODB01(oracle)-ssh-keygen -t dsa #生成dsa公钥 (直接回车不需要输入密码)ODB01(oracle)-ssh-keygen -t rsa #生成dsa公钥(直接回车不需要输入密码)ODB01：ODB01(oracle)-cat /.ssh/id_rsa.pub /.ssh/authorized_keysODB01(oracle)-cat /.ssh/id_dsa.pub /.ssh/authorized_keys ODB01(oracle)-scp /.ssh/authorized_keys 43:/home/oracle/.ssh/ #(输入ODB02密码，即可复制公钥到ODB02节点上)ODB02：ODB01(oracle)-cat /.ssh/id_rsa.pub /.ssh/authorized_keysODB01(oracle)-cat /.ssh/id_dsa.pub /.ssh/authorized_keys ODB01(oracle)-scp /.ssh/authorized_keys 42:/home/oracle/.ssh/ #(输入ODB01密码，即可复制到ODB02节点上)3.10禁用时间同步NTPD服务1) ODB01节点禁止NTP ServerrootODB01 # mv /etc/ntp.conf /etc/ntp_bak.confrootODB01 #service ntpd stoprootODB01 #chkconfig ntpd off #禁止系统重启加载1) ODB02节点禁止NTP ServerrootODB02# mv ntp.conf ntp_bak.confrootODB02 #service ntpd stoprootODB02#chkconfig ntpd off #禁止系统重启加载3.11安装并配置 ASM驱动检查操作系统版本，确认安装不操作系统相匹配的 asmlib，在所有节点执行。安装asmlib之前需要用命令检查当前系统版本，并下载对应内核的版本的asmlib版本。3.11.1安装asmlib按照下列顺序安装asmlib（在所有节点安装） rpm ivh oracleasm-support-2.1.8-1.el5.x86_64.rpmrpm ivh oracleasm-2.6.18-308.el5xen-2.0.5-1.el5.x86_64.rpmrpm ivh oracleasm-2.6.18-308.el5-debuginfo-2.0.5-1.el5.x86_64.rpmrpm ivh oracleasmlib-2.0.4-1.el5.x86_64.rpmrpm ivh oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm3.11.2初始化Asmlib初始化Asmlib,在所有节点执行rootODB01 asm# /usr/sbin/oracleasm configure -iConfiguring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets (). Hitting without typing ananswer will keep that current value. Ctrl-C will abort.Default user to own the driver interface : gridDefault group to own the driver interface : asmadminStart Oracle ASM library driver on boot (y/n) n: yScan for Oracle ASM disks on boot (y/n) y: yWriting Oracle ASM library driver configuration: done3.11.3将asmlib加载到内核将asmlib加载到内核（在所有节点执行）rootODB01 asm# /usr/sbin/oracleasm initCreating /dev/oracleasm mount point: /dev/oracleasmLoading module oracleasm: oracleasm Mounting ASMlib driver filesystem: /dev/oracleasm 3.12磁盘分区Oracle RAC环境搭建存储管理采用Oracle自劢存储管理技术ASM。系统磁盘分为本地磁盘和共享磁盘。共享磁盘由EMC存储映射给两个节点。ASM自劢存储管理使用的共享磁盘，磁盘分区也是对共享磁盘进行分区。Oracle RAC环境搭建ASM共使用三个分区，分区详细信息，如下：分区所在磁盘ASM名称大小大小/dev/sdbVOTDISK5G仲裁判/dev/sdcDATA1200G数据库/dev/sddDATA2200G数据库/dev/sdeFLASH100G备份3.13创建ASM磁盘创建ASM磁盘（在一个节点执行）rootODB01 asm# /usr/sbin/oracleasm createdisk VOTDISK /dev/emcpowera1Writing disk header: doneInstantiating disk: donerootODB01 asm# /usr/sbin/oracleasm createdisk DATADISK1/dev/emcpowerb1Writing disk header: doneInstantiating disk: donerootODB01 asm# /usr/sbin/oracleasm createdisk DATADISK2 /dev/emcpowerc1Writing disk header: doneInstantiating disk: donerootODB01 asm# /usr/sbin/oracleasm createdisk FLASHDISK /dev/emcpowerd1Writing disk header: doneInstantiating disk: donerootODB02asm# /usr/sbin/oracleasm listdisksDATADISKFLASHDISKVOTDISKODB02执行扫描rootODB02 # /usr/sbin/oracleasm scandisks Reloading disk partitions: done Reloading disk partitions: done Reloading disk partitions: done Instantiating disk DATADISK Instantiating