[计算机硬件及网络]华为服务器Linux55+Oracle11gRAC.doc

本文记录了在两台华为Tecal RH2488 V2服务器（Red Hat Enterprise Linux Server release 5.5 (Tikanga)）上实现部署Oracle 11g RAC（ASM）的全过程。一、服务器环境准备1.1服务器环境规划下面说一下我的规划，磁盘空间总共做下来大概占用空间40G左右，内存的话最好每个节点尽量多分配一点节点名称: PYDB1 , PYDB2 操作系统硬盘空间每节点至少20G数据库名称： pycgdb内存分配： 每节点 64GBIP及SCANIP : Public IP: 0, 3Private IP: 1, 2VIP: 1 , 4SCANIP: 5打开hosts文件，追加以下信息：#public0 PYDB13 PYDB2#public-vip1 PYDB1-vip4 PYDB2-vip#private1 PYDB1-priv2 PYDB2-priv#scan 5 scan-clusterPYDB1:Eth0 和eth1绑定成bond0 0Eth4 和eth5绑定成bond1 1PYDB2:Eth0 和eth1绑定成bond0 3Eth4 和eth5绑定成bond1 21.2磁盘空间规划安装操作系统和数据库软件所磁盘空间至少为8G，建议使用HP BL680刀片服务器自带的300G磁盘。由于我们准备采用asm来创建rac，将voting disk和ocr放在asm上，数据文件等放在asm上其中voting disk与ocr需要至少20M和60M的空间，建议用3个大小为2G的ASM磁盘(OCR_VOTE01、OCR_VOTE02、OCR_VOTE03) 创建1个NORMAL冗余的磁盘组： +OCR_VOTE数据文件等至少需要12G的空间，建议用4个大小为300G的ASM磁盘(ASM_DATA01、ASM_DATA02、ASM_DATA03、ASM_DATA04) 创建1个EXTERNAL冗余的磁盘组： +ASM_DATA归档日志文件等至少需要12G的空间，建议用1个大小为300G的ASM磁盘(ASM_ARCH01) 创建1个EXTERNAL冗余的磁盘组： +ASM_ARCH1.3 开启telnet/ftp服务在安装好Redhat以后，需要设置Ftp和Telnet服务文件，才能启动Ftp和Telnet服务，可以通过远程控制进行开启。1.3.1、用root用户登录，进入终端命令模式；1.3.2、开启ftp服务。cd /etc/xinetd.d ，编辑ftp服务的配置文件gssftp的设置：rootPYDB1 vmware-tools-distrib# cd /etc/xinetd.drootPYDB1 xinetd.d# vi gssftp# default: off# description: The kerberized FTP server accepts FTP connections # that can be authenticated with Kerberos 5.service ftp flags = REUSE socket_type = stream wait = no user = root server = /usr/kerberos/sbin/ftpd server_args = -l log_on_failure += USERID disable = no图1 开启ftp服务设置.vi gssftp ，将 修改两项内容：（1）server_args = -l a 去掉-a 改为server_args = -l（2）disable=yes改为disable=no（3）保存退出。1.3.3、开启telnet服务cd /etc/xinetd.d ，察看telnet服务的配置文件krb5-telnet的设置：rootPYDB1 xinetd.d# vi krb5-telnet# default: off# description: The kerberized telnet server accepts normal telnet sessions, # but can also use Kerberos 5 authentication.service telnet flags = REUSE socket_type = stream wait = no user = root server = /usr/kerberos/sbin/telnetd log_on_failure += USERID disable = no图2 开启telnet服务设置vi krb5-telnet ,（1）将disable=yes,改为disable=no,保存退出。1.3.4、激活服务：telnet/ftp是挂在xinetd底下的，所以自然只要重新激活 xinetd 就能够将xinetd 里头的设定重新读进来，所以刚刚设定的telnet/ftp自然也就可以被激活。激活目录和命令：cd /etc/rc.d/init.d/service xinetd restart有时会提示命令不存在，需要加上命令的路径rootPYDB1 init.d# service xinetd resartbash: service: command not foundrootPYDB1 init.d# /sbin/service xinetd restart图3 重新启动FTP、Telnet服务1.3.5连上FTP图4 1.4 为 OracleRAC 准备共享存储在三个分开(三个物理磁盘)的 failuregroups 下需要至少 2GB 的磁盘空间用于OracleClusterware文件 (OracleClusterRegistryandvotingdisk).AllofthedevicesinanAutomaticStorageManagementdiskgroupshouldbethesamesizeandhavethesameperformancecharacteristics.Adiskgroupshouldnotcontainmorethanonepartitiononasinglephysicaldiskdevice.UsinglogicalvolumesasadeviceinanAutomaticStorageManagementdiskgroupisnotsupportedwithOracleRAC.Theuseraccountwithwhichyouperformtheinstallation(oracle)musthavewritepermissionstocreatethefilesinthepaththatyouspecify.1.4.1共享磁盘划分规划(硬件上已做 RAID01，这里规划镜像磁盘组)BlockDevice ASMlibName Size Comments/dev/mapper/mpath4p1 OCR_VOTE01 2GB ASMDiskgroupforOCRandVotingDisks/dev/mapper/mpath5p1 OCR_VOTE02 2GB ASMDiskgroupforOCRandVotingDisks/dev/mapper/mpath6p1 OCR_VOTE03 2GB ASMDiskgroupforOCRandVotingDisks/dev/mapper/mpath0p1 ASM_DATA01 300GB ASMDataDiskgroup/dev/mapper/mpath1p1 ASM_DATA02 300GB ASMDataDiskgroup （镜像磁盘组）/dev/mapper/mpath2p1 ASM_DATA03 300GB ASMData Diskgroup/dev/mapper/mpath3p1 ASM_ DATA04 300GB ASMData Diskgroup （镜像磁盘组）/dev/mapper/mpath7p1 ASM_ARCH01 300GB ASMArchiveDiskgroup1.4.2san存储多路径软件的安装及配置1，san存储多路径软件的安装链路聚合需要的2个软件分别在安装光盘的Server目录下面device-mapper-1.02.39-1.el5.x86_64device-mapper-multipath-0.4.7-34.el5.x86_64device-mapper-event-1.02.39-1.el5.x86_64device-mapper-1.02.39-1.el5.i386安装完毕后应该包括下面的软件包rootPYDB1 # rpm -qa|grep devicedevice-mapper-1.02.39-1.el5device-mapper-1.02.39-1.el5device-mapper-event-1.02.39-1.el5device-mapper-multipath-0.4.7-34.el52，san存储多路径软件的配置1、使用 ntsysv 命令将mulitipath服务开启* multipathd 2、启动该服务rootmail init.d# service multipathd startStarting multipathd daemon: 3、修改配置文件/etc/multipath.conf # This is a basic configuration file with some examples, for device mapper# multipath.# For a complete list of the default configuration values, see# /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.defaults# For a list of configuration options with descriptions, see# /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.annotated# Blacklist all devices by default. Remove this to enable multipathing# on the default devices. blacklist devnode *# By default, devices with vendor = IBM and product = S/390.* are# blacklisted. To enable mulitpathing on these devies, uncomment the# following lines.#blacklist_exceptions #device #vendorIBM#productS/390.*# Use user friendly names, instead of using WWIDs as names.defaults user_friendly_names yes# Here is an example of how to configure some standard options.#defaults udev_dir/devpolling_interval 10selectorround-robin 0path_grouping_policymultibusgetuid_callout/sbin/scsi_id -g -u -s /block/%nprio_callout/bin/truepath_checkerreadsector0rr_min_io100max_fds8192rr_weightprioritiesfailbackimmediateno_path_retryfailuser_friendly_namesyes# The wwid line in the following blacklist section is shown as an example# of how to blacklist devices by wwid. The 2 devnode lines are the# compiled in default blacklist. If you want to blacklist entire types# of devices, such as all scsi devices, you should use a devnode line.# However, if you want to blacklist specific devices, you should use# a wwid line. Since there is no guarantee that a specific device will# not change names on reboot (from /dev/sda to /dev/sdb for example)# devnode lines are not recommended for blacklisting specific devices.#blacklist wwid 26353900f02796769devnode (ram|raw|loop|fd|md|dm-|sr|scd|st)0-9*devnode hda-zmultipaths #multipath #wwid3600508b4000156d700012000000b0000#aliasyellow#path_grouping_policymultibus#path_checkerreadsector0#path_selectorround-robin 0#failbackmanual#rr_weightpriorities#no_path_retry5#multipath #wwid1DEC_321816758474#aliasred#devices device vendorEMCproductVNX5500path_grouping_policymultibusgetuid_callout /sbin/scsi_id -g -u -s /block/%npath_checkerreadsector0path_selectorround-robin 0hardware_handler0failback15rr_weightprioritiesno_path_retryqueue#device #vendorCOMPAQ #productMSA1000 #path_grouping_policymultibus#4、使用 multipath -F 删除现有路径 5、multipath -v2 格式化路径6、使用 multipath -ll 查看多路径rootPYDB1 # multipath -llmpath2 (36006016059813100182e2f3c8012e211) dm-7 DGC,RAID 10size=300Gfeatures=1 queue_if_no_pathhwhandler=1 emcrw_ round-robin 0 prio=2active _ 0:0:0:7 sdh 8:112 activeready _ 1:0:0:7 sdx 65:112 activeready_ round-robin 0 prio=0enabled _ 1:0:1:7 sdaf 65:240 activeready _ 0:0:1:7 sdp 8:240 activereadympath1 (36006016059813100afcaeb298012e211) dm-4 DGC,RAID 10size=300Gfeatures=1 queue_if_no_pathhwhandler=1 emcrw_ round-robin 0 prio=2active _ 1:0:1:4 sdac 65:192 activeready _ 0:0:1:4 sdm 8:192 activeready_ round-robin 0 prio=0enabled _ 0:0:0:4 sde 8:64 activeready _ 1:0:0:4 sdu 65:64 activereadympath0 (36006016059813100b1caeb298012e211) dm-6 DGC,RAID 10size=300Gfeatures=1 queue_if_no_pathhwhandler=1 emcrw_ round-robin 0 prio=2active _ 1:0:1:6 sdae 65:224 activeready _ 0:0:1:6 sdo 8:224 activeready_ round-robin 0 prio=0enabled _ 0:0:0:6 sdg 8:96 activeready _ 1:0:0:6 sdw 65:96 activereadympath7 (36006016059813100aecaeb298012e211) dm-3 DGC,RAID 10size=300Gfeatures=1 queue_if_no_pathhwhandler=1 emcrw_ round-robin 0 prio=2active _ 0:0:0:3 sdd 8:48 activeready _ 1:0:0:3 sdt 65:48 activeready_ round-robin 0 prio=0enabled _ 1:0:1:3 sdab 65:176 activeready _ 0:0:1:3 sdl 8:176 activereadympath6 (360060160598131002c7cdffc7f12e211) dm-2 DGC,RAID 10size=2.0Gfeatures=1 queue_if_no_pathhwhandler=1 emcrw_ round-robin 0 prio=2active _ 0:0:0:2 sdc 8:32 activeready _ 1:0:0:2 sds 65:32 activeready_ round-robin 0 prio=0enabled _ 1:0:1:2 sdaa 65:160 activeready _ 0:0:1:2 sdk 8:160 activereadympath5 (36006016059813100e476f8e07f12e211) dm-9 DGC,RAID 10size=2.0Gfeatures=1 queue_if_no_pathhwhandler=1 emcrw_ round-robin 0 prio=2active _ 0:0:0:1 sdb 8:16 activeready _ 1:0:0:1 sdr 65:16 activeready_ round-robin 0 prio=0enabled _ 0:0:1:1 sdj 8:144 activeready _ 1:0:1:1 sdz 65:144 activereadympath4 (360060160598131009ae6ce997f12e211) dm-8 DGC,RAID 10size=2.0Gfeatures=1 queue_if_no_pathhwhandler=1 emcrw_ round-robin 0 prio=2active _ 0:0:0:0 sda 8:0 activeready _ 1:0:0:0 sdq 65:0 activeready_ round-robin 0 prio=0enabled _ 0:0:1:0 sdi 8:128 activeready _ 1:0:1:0 sdy 65:128 activereadympath3 (36006016059813100b0caeb298012e211) dm-5 DGC,RAID 10size=300Gfeatures=1 queue_if_no_pathhwhandler=1 emcrw_ round-robin 0 prio=2active _ 0:0:0:5 sdf 8:80 activeready _ 1:0:0:5 sdv 65:80 activeready_ round-robin 0 prio=0enabled _ 1:0:1:5 sdad 65:208 activeready _ 0:0:1:5 sdn 8:208 activereadyrootPYDB1 #7、使用fdisk -l 查看多路径的情况 rootPYDB1 # fdisk -lDisk /dev/cciss/c0d0: 299.9 GB, 299966445568 bytes255 heads, 63 sectors/track, 36468 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/cciss/c0d0p1 * 1 13 104391 83 Linux/dev/cciss/c0d0p2 14 36468 292824787+ 8e Linux LVMDisk /dev/sda: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sda1 1 261 2096451 83 LinuxDisk /dev/sdb: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdb1 1 261 2096451 83 LinuxDisk /dev/sdc: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdc1 1 261 2096451 83 LinuxDisk /dev/sdd: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdd1 1 39162 314568733+ 83 LinuxDisk /dev/sdf: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdf1 1 39162 314568733+ 83 LinuxDisk /dev/sdh: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdh1 1 39162 314568733+ 83 LinuxDisk /dev/sdm: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdm1 1 39162 314568733+ 83 LinuxDisk /dev/sdo: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdo1 1 39162 314568733+ 83 LinuxDisk /dev/sdq: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdq1 1 261 2096451 83 LinuxDisk /dev/sdr: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdr1 1 261 2096451 83 LinuxDisk /dev/sds: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sds1 1 261 2096451 83 LinuxDisk /dev/sdt: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdt1 1 39162 314568733+ 83 LinuxDisk /dev/sdv: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdv1 1 39162 314568733+ 83 LinuxDisk /dev/sdx: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdx1 1 39162 314568733+ 83 LinuxDisk /dev/sdac: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdac1 1 39162 314568733+ 83 LinuxDisk /dev/sdae: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/sdae1 1 39162 314568733+ 83 LinuxDisk /dev/dm-2: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/dm-2p1 1 261 2096451 83 LinuxDisk /dev/dm-3: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/dm-3p1 1 39162 314568733+ 83 LinuxDisk /dev/dm-4: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/dm-4p1 1 39162 314568733+ 83 LinuxDisk /dev/dm-5: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/dm-5p1 1 39162 314568733+ 83 LinuxDisk /dev/dm-6: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/dm-6p1 1 39162 314568733+ 83 LinuxDisk /dev/dm-7: 322.1 GB, 322122547200 bytes255 heads, 63 sectors/track, 39162 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/dm-7p1 1 39162 314568733+ 83 LinuxDisk /dev/dm-8: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/dm-8p1 1 261 2096451 83 LinuxDisk /dev/dm-9: 2147 MB, 2147483648 bytes255 heads, 63 sectors/track, 261 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/dm-9p1 1 261 2096451 83 LinuxDisk /dev/dm-10: 322.1 GB, 322118383104 bytes255 heads, 63 sectors/track, 39161 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/dm-10 doesnt contain a valid partition tableDisk /dev/dm-11: 322.1 GB, 322118383104 bytes255 heads, 63 sectors/track, 39161 cylindersUnits = cylinders