Linux下vcs安装及oracle部署-裸设备版本.doc

Linux下vcs安装及oracle部署-裸设备版本北京握奇数据系统有限公司文件状态： 草稿 修改稿 正式发布文档密级：当前版本：部 门： cc项目名称： 作 者：张建双完成日期：审 核：日 期：审 批：日 期： Linux下vcs安装及oracle部署-裸设备版本版本历史版本日期作者说明102011-06-15张建双1.12011-07-13张建双根据测试人员建议修改部分描述模糊的词语增加第9章节配置应用程序资源组第67页 总67页目录1引言51.1文档目的51.2读者对象51.3约定51.4术语与缩写解释52安装前准备52.1支持的linux操作系统5VCS 支持的卷管理及文件系统62.2软硬件准备及检查72.3修改/etc/hosts文件72.4建立双机信任机制73VCS软件安装94配置vcs125验证VCS165.1验证LLT和GAB的配置文件165.2验证LLT，GAB和Cluster操作176创建VG及逻辑卷197创建双机数据库-裸设备方式227.1创建数据库227.2设置用户密码257.3使用裸设备文件267.4不指定恢复区267.5不勾选任何特性277.6设置memory287.7选择数据库字符集297.8指定初始化参数文件位置297.9修改控制文件的路径317.10修改控制文件记录317.11配置数据文件327.12配置redo347.13完成数据库的创建347.14将安装好的数据库软件及数据库copy到备机367.15新增加表空间或数据文件的维护378配置Oracle资源组388.1安装VCS agent for Oracle388.2修改listener.ora文件 （以oracle用户登录）408.3导入双机管理脚本408.4创建service group418.5增加资源468.6建立依赖关系498.7启动资源519配置应用程序资源组539.1创建用户539.2署产品包539.3查看系统启动脚本579.4配置和查看系统日志579.5备机环境拷贝589.6创建service group5910VCS的卸载641 引言1.1 文档目的文档详细介绍VCS的安装过程已经如何配置oracle资源，针对裸设备配置数据库方面。1.2 读者对象技术支持、开发人员及系统维护人员1.3 约定文档中无特殊说明，所进行的操作都以root用户。如果需要oracle或其他业务用户会在操作前说明。1.4 术语与缩写解释术语解释VCSVeritas Cluster Server 集群软件2 安装前准备2.1 支持的linux操作系统This section lists the supported operating systems for this release of Veritasproducts.For important updates regarding this release, review the Late-Breaking NewsTechNote on the Symantec Technical Support website:/docs/335001The Veritas 5.1 SP1 release operates on the following operating systems andhardware: Red Hat Enterprise Linux 5 (RHEL 5) with Update 3 (2.6.18-128.el5 kernel) orlater on AMD Opteron or Intel Xeon EM64T (x86_64) SUSE Linux Enterprise Server 10 (SLES 10) with SP2 (0-0.21 kernel)or SP3 on AMD Opteron or Intel Xeon EM64T (x86_64) SUSE Linux Enterprise Server 11 (SLES 11) (9-5-default kernel) orSUSE Linux Enterprise Server 11 (SLES 11) with SP1 on AMD Opteron or IntelXeon EM64T (x86_64) Oracle Enterprise Linux (OEL 5) with Update 3 or later需要的rpm包操作系统需要的rpm包RHEL 5compat-libstdc+-33-3.2.3-61.x86_64 rpmglibc-2.5-42.i686 rpmglibc-2.5-42.x86_64 rpmksh-20080202-14.el5.x86_64 rpmlibgcc-4.1.2-46.el5.i386 rpmlibgcc-4.1.2-46.el5.x86_64 rpmlibstdc+-4.1.2-46.el5.i386 rpmpam--6.el5.x86_64 rpmSLES 10compat-libstdc+-5.0.7-22.2.x86_64 rpmglibc-2.4-31.54.x86_64 rpmglibc-32bit-2.4-31.54.x86_64 rpmksh-93s-59.7.x86_64 rpmlibgcc-4.1.2_20070115-0.21.x86_64 rpmlibstdc+-4.1.2_20070115-0.21.x86_64rpmpam--28.13.x86_64 rpmSLES 11glibc-2.9-13.2.x86_64glibc-32bit-2.9-13.2.x86_64 rpmksh-93t-9.4.x86_64 rpmlibgcc43-32bit-4.3.3_20081022-11.18.x86_64 rpmlibgcc43-4.3.3_20081022-11.18.x86_64 rpmlibstdc+33-3.3.3-11.9.x86_64 rpmlibstdc+43-32bit-4.3.3_20081022-11.18.x86_64 rpmVCS 支持的卷管理及文件系统 ext2, ext3, reiserfs, NFS, and bind on LVM2, raw disks, and VxVM. Veritas Storage Foundation (SF): Veritas Volume Manager (VxVM) with VeritasFile System (VxFS)2.2 软硬件准备及检查软硬件要求：需要两台服务器，一台存储设备，交换机一台，网卡2个以上，最好至少3个）服务器配置：需要安装同一版本企业版本Linux 系统，服务器最少配置3块网卡（心跳网卡，数据网卡），连接磁盘阵列所需要的HBA 卡。磁盘阵列配置：要求把磁盘阵列的同一LUN 分别映射给服务器A,服务器B,并确认每台服务器都可以正常识别到同一分区VCS软件一套（需要license）软硬件检查：1：确保主机与磁盘阵列，网卡物理连接正确无误2: 确保两台主机均能访问到相同的共享磁盘设备，3：应用服务或数据库不能配置为开机自启动4: 在配置和使用过程中，确保同时只有一台主机挂载共享磁盘；切忌两台主机同时挂载共享磁盘。共享磁盘不能配置为操作系统启动时自动挂载2.3 修改/etc/hosts文件本文档两台主机名定义为Mainha,standbyha。系统安装后首先编辑两台主机的hosts文件，列出两台主机及浮动IP 的名字及其对应名字。一定要在两台主机上都先编辑好主机表。 注意：本操作要在两台主机上完成rootmainha vg# vi /etc/hosts 红色的三行为新增加# Do not remove the following line, or various programs# that require network functionality will fail. mainha localhost.localdomain localhost:1 localhost6.localdomain6 localhost6 mainha standbyha4 vip2.4 建立双机信任机制VCS 安装和配置需要使用rsh 或者ssh，配置成不需要密码访问。目前我们都采用ssh 方式。首先在mainha上生成私钥和公钥，拷贝公钥到standbyha 上。红色为输入，Enter代表不需要输入任何值，直接回车确认。另外如果你已经做过，但没有成功，再次按照此步骤做时，会需要覆盖原先的.ssh 目录下的文件，回答y 确认-创建公钥、私钥rootmainha# ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): EnterCreated directory /root/.ssh.Enter passphrase (empty for no passphrase): EnterEnter same passphrase again: EnterYour identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:59:20:a2:7b:7f:64:be:94:99:74:e5:a3:b9:6b:f3:fe rootmainha-创建standbyha的/root/.ssh 目录rootmainha# ssh standbyha mkdir /root/.sshThe authenticity of host standbyha (1) cant be established.RSA key fingerprint is 39:65:39:35:ff:8f:29:f1:18:94:09:8b:c7:48:0c:d0.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added standbyha,1 (RSA) to the list of known hosts.rootstandbyhas password:wa1234 -root 用户密码-拷贝公钥rootmainha # scp /root/.ssh/id_rsa.pub standbyha:/root/.ssh/authorized_keysrootstandbyhas password: wa1234 -root 用户密码id_rsa.pub 100% 395 0.4KB/s 00:00-使用下列命令测试， #ssh standbyha ls /root 不需要口令即可访问测试成功后使用相同的办法，在standbyha上生成私钥和公钥，拷贝公钥到mainha上，达到双向都能不输口令访问注意：这步如果不成功，不能继续做下一节的操作3 VCS软件安装本文档以Veritas Cluster Server 5.1 SP1为例介绍VCS的安装过程，其他版本的可能和此版本有细微的差别，但是方法是一样的进入vcs安装文件所在目录rootmainha cluster_server# pwd/root/dvd1-redhatlinux/rhel5_x86_64/cluster_server 解压缩后的路径rootmainha cluster_server# ./installvcs install -下面开始安装rootmainha cluster_server# ./installvcs -install Veritas Cluster Server 5.1 SP1 Install Logs are being written to /var/tmp/installvcs-201105121609eSY while installvcs is in progress.Do you agree with the terms of the End User License Agreement as specified in the cluster_server/EULA/en/EULA_VCS_Ux_5.1SP1.pdf file present on media? y,n,q,? 输入y-询问你是否读了license内容 Veritas Cluster Server 5.1 SP1 Install Program 1) Install minimal required Veritas Cluster Server rpms - 272 MB required 2) Install recommended Veritas Cluster Server rpms - 306 MB required 3) Install all Veritas Cluster Server rpms - 341 MB required 4) Display rpms to be installed for each optionSelect the rpms to be installed on all systems? 1-4,q,? (2) enter即可选择安装类型Enter the 64 bit RHEL5 system names separated by spaces: q,? (mainha standbyha) mainha standbyha 输入要安装配置vcs的节点的名称（主机名）这里vcs会检查这些机器之间的通讯，如前面的ssh配置是否正确，检查这些操作系统是否满足vcs安装需求检查操作系统中是否有vcs服务需要的rpms包检查vcs服务需要的空间等当出现类似下面的内容时表示系统检查是通过了，System verification checks completed successfullyThe following warnings were discovered on the systems:-警告信息Nodes have difference in clock by more than 5 sec Do you want to continue? y,n,q (y) y -输入y继续安装The following Veritas Cluster Server rpms will be installed on all systems:Rpm Rpm DescriptionVRTSvlic Veritas LicensingVRTSperl Veritas Perl 5.10.0 RedistributionVRTSspt Veritas Software Support Tools by SymantecVRTSatClient Symantec Product Authentication Service ClientVRTSatServer Symantec Product Authentication ServiceVRTSllt Veritas Low Latency TransportVRTSgab Veritas Group Membership and Atomic BroadcastVRTSvxfen Veritas I/O Fencing by SymantecVRTSamf Veritas Asynchronous Monitoring Framework by SymantecVRTSvcs Veritas Cluster ServerVRTSvcsag Veritas Cluster Server Bundled Agents by SymantecVRTSvcsdr Veritas Cluster Server Disk Reservation ModulesVRTSvcsea Veritas Cluster Server Enterprise Agents by SymantecPress Enter to continue: -列出安装内容列表，按enter继续安装成功如果是商用环境，这里会提示输入license，如下图1) Enter a valid license key2) Enable keyless licensing and complete system licensing laterHow would you like to license the systems? 1-2,q (2)VCS is licensed on all systems. Do you wish to enter additional licenses? y,n,q (n)是否要键入附加key，如果需要配置全局clusters需要选择y，（我们应该不需要都是本地clusters 因此我们输入n）Would you like to configure VCS on mainha standbyha? y,n,q (n) n 是否在每个节点上配置vcs，我们选择n，配置将在下一步完成。4 配置vcsrootmainha cluster_server# ./installvcs configure -执行该命令进行配置Enter the 64 bit RHEL5 system names separated by spaces: q,? (mainha standbyha) mainha standby-和安装一样，输入需要配置的节点名称System verification checks completed successfullyThe following warnings were discovered on the systems:Nodes have difference in clock by more than 5 secDo you want to continue? y,n,q (y) y 检查发现节点的时钟差别超过5秒，这里忽略这个警告，不过尽量将两个系统的时间设置到一样I/O FencingA split brain can occur if servers within the cluster become unable to communicate for any number of reasons. If I/O Fencing is notenabled, you run the risk of data corruption should a split brain occur. Therefore, to avoid data corruption due to split brain in CFSenvironments, I/O Fencing has to be enabled.Do you want to configure I/O Fencing in enabled mode? y,n,q,? (y) n-这里提示我们是否配置I/O Fencing，这个功能用来保护共享存储数据不会被并发访问，但是需要安装其他软件来支持这个选项，我们这里选择nConfiguring the cluster name and IDPress Enter to continue: -提示信息之间enterEnter the unique cluster name: q,? cletest - 给Cluster 命名Enter a unique cluster ID number between 0-65535: b,q,? 8-ClusterID 数字随意定，但同网络内VCS 不能有相同的数字，否则会导致LLT 无法运行Configuring private heartbeat links1) Configure heartbeat links using LLT over Ethernet 2) Configure heartbeat links using LLT over UDP 3) Automatically detect configuration for LLT over Ethernet b) Back to previous menuHow would you like to configure heartbeat links? 1-3,b,q,? (1)-Symantec建议使用 LLT over Ethernet，所以我们选择1Discovering NICs on mainha . Discovered eth0 eth1 sit0 发现的网卡设备Enter the NIC for the first private heartbeat link on mainha: b,q,? (sit0) eth1 eth1作为第一个心跳线，正式系统上应该配置2个心跳线，这里测试服务器只有一个网卡作为心跳线Are you sure you want to use eth1 for the first private heartbeat link? y,n,q,b,? (n) yWould you like to configure a second private heartbeat link? y,n,q,b,? (y)正式系统上应该配置2个心跳线，这里测试服务器只有一个网卡，所以选择了n ，正式系统选择y说明：SIT是IPv6-in-IPv4 tunnel interfaces, 这个interfaces(界面或是接口)也称作sitx, sit是Simple Internet Transition的缩写. 它可以将IPv6的数据包塞进IPv4, 通过IPv4到达另一个地点.sit0 不能使用在专用的tunnels 上.Enter the NIC for the low-priority heartbeat link on mainha: b,q,? (eth0) 这里由于只配置了一个心跳线，所以强制要求配置一个低优先级的心跳，而实际生产环境中由于配置了两个心跳网卡，因此不需要配置low-priority heartbeat 这里选择n即可 ，实际生产环境的配置参照下面Are you using the same NICs for private heartbeat links on all systems? y,n,q,b,? (y) y 是否确认所有节点上都使用同样的网卡设备名（如每个节点上都用eth1、eth2作为心跳线），选择y ，这样能够统一好管理Cluster information verification: Cluster Name: cletest Cluster ID Number: 8 Private Heartbeat NICs for mainha: link1=eth1 Low-Priority Heartbeat NIC for mainha: link-lowpri1=eth0 Private Heartbeat NICs for standbyha: link1=eth1 Low-Priority Heartbeat NIC for standbyha: link-lowpri1=eth0Is this information correct? y,n,q,b,? (y) 确认信息是否正确，直接回车Configuring the virtual IP of the cluster You can configure the virtual IP of the cluster to use to connect to the ClusterManager (Java Console) or to specify in the RemoteGroup resource. Do you want to configure the Virtual IP? y,n,q,? (n) n -是否通过虚拟ip来连接和管理集群软件，我们选择nWould you like to configure VCS to use Symantec Security Services? y,n,q (n) -是否需要配置Security Services 选择n跳过配置用户Do you want to set the username and/or password for the Admin user(default username = admin, password=password)? y,n,q (y) 是否需要配置admin用户或增加其他用户Do you want to set the username and/or password for the Admin user(default username = admin, password=password)? y,n,q (n) yEnter the user name: b,q,? (admin) adminEnter the password: Enter again: Do you want to add another user to the cluster? y,n,q (n) y 创建另一个用户Enter the user name: b,q,? zhangjs -这个是备用用户，自行定义，暂时还没有用到Enter the password: Enter again:Enter the privilege for user zhangjs (A=Administrator, O=Operator, G=Guest): b,q,? oWould you like to add another user? y,n,q (n) nVCS User verification: User: admin Privilege: Administrators User: zhangjs Privilege: Operators Passwords are not displayedIs this information correct? y,n,q (y) y-验证信息是否正确，输入y回车Do you want to configure SMTP notification? y,n,q,? (n) n 配置smtp通知，现在nDo you want to configure SNMP notification? y,n,q,? (n) n All VCS processes that are currently running must be stoppedDo you want to stop VCS processes now? y,n,q,? (y) y 到此Vcs的安装配置完成5 验证VCS说明：验证工作要在两个节点上分别操作5.1 验证LLT和GAB的配置文件验证前将/opt/VRTS/bin分别配置到/etc/profile 或者root 用户的配置文件例如：rootmainha # vi /etc/profile-增加下面的内容到文件末尾PATH=$PATH:/opt/VRTS/bin1、验证LLT配置文件：安装之后，/etc/llthosts和/etc/llttab文件会包含LLT的配置信息，类似信息如下所示（分别在各个节点验证）。rootmainha # more /etc/llthosts0 mainha1 standbyharootstandbyha # more /etc/llthosts0 mainha1 standbyharootstandbyha #第一列为主机在集群中的系统ID，第二列为主机在集群中的主机名rootmainha # more /etc/llttabset-node mainhaset-cluster 8link eth1 eth-00:1a:64:69:df:e2 - ether - -link-lowpri eth0 eth-00:1a:64:69:df:e0 - ether - -rootstandbyha # more /etc/llttabset-node standbyhaset-cluster 8link eth1 eth-00:1a:64:c2:44:ca - ether - -link-lowpri eth0 eth-00:1a:64:c2:44:c8 - ether - -rootstandbyha #/etc/llttab文件中，第一行标识为主机名；第二行标识为集群唯一标识；第三行和第四行为本机LLT协议使用的网卡名。2、验证GAB配置文件：两台主机安装VCS之后，/etc/gabtab文件会包含GAB的配置信息，类似信息如下所示。 rootmainha # more /etc/gabtab/sbin/gabconfig -c -n2rootstandbyha # more /etc/gabtab/sbin/gabconfig -c -n25.2 验证LLT，GAB和Cluster操作验证LLT运行状态使用lltstat nvv命令，查看两台主机上LLT的运行状态，类似输出信息如下(只显示部分)：Mainha主机：rootmainha # lltstat -nvvLLT node information: Node State Link Status Address * 0 mainha OPEN eth1 UP 00:1A:64:69:DF:E2 eth0 UP 00:1A:64:69:DF:E0 1 standbyha OPEN eth1 UP 00:1A:64:C2:44:CA eth0 UP 00:1A:64:C2:44:C8 2 CONNWAIT eth1 DOWN eth0 DOWN 3 CONNWAIT eth1 DOWN standbyharootstandbyha # lltstat -nvvLLT node information: Node State Link Status Address 0 mainha OPEN eth1 UP 00:1A:64:69:DF:E2 eth0 UP 00:1A:64:69:DF:E0 * 1 standbyha OPEN eth1 UP 00:1A:64:C2:44:CA eth0 UP 00:1A:64:C2:44:C8 2 CONNWAIT eth1 DOWN eth0 DOWN CONNWAIT验证GAB运行状态使用gabconfig a命令，查看两台主机上GAB的运行状态，类似输出信息如下:rootmainha # gabconfig -aGAB Port Memberships=Port a gen 87be01 membership 01Port h gen 87be04 membership 01 rootstandbyha # gabconfig -aGAB Port Memberships=Port a gen 87be01 membership 01Port h gen 87be04 membership 01 Port a 表示GAB正在通信，membership 01表示集群中系统标识为0和1的主机是处于连接状态。Port h 表示 VCS已经起来，membership 01表示集群中系统标识为0和1的主机的VCS处于运行状态。验证集群操作使用hastatus -summary命令，查看两台主机上VCS的运行状态，类似输出信息如下：rootmainha # hastatus -summary- SYSTEM STATE- System State Frozen A mainha RUNNING 0 A standbyha RUNNING 0 rootstandbyha # hastatus -summary- SYSTEM STATE- System State Frozen A mainha RUNNING 0 A standbyha RUNNING 0State 为RUNNING ，VCS一切正常。6 创建VG及逻辑卷l 使用fdisk l查看所有存储分配的lun# fdisk -l |grep Diskrootstandbyha # fdisk -l |grep DiskDisk /dev/sdc doesnt contain a valid partition tableDisk /dev/sde doesnt contain a valid partition tableDisk /dev/sdf doesnt contain a valid partition tableDisk /dev/sda: 73.2 GB, 73284976640 bytesDisk /dev/sdb: 293.3 GB, 293378981888 bytesDisk /dev/sdc: 524 MB, 524288000 bytesDisk /dev/sdd: 161.0 GB, 161061273600 bytes/磁盘阵列Disk /dev/sde: 107.3 GB, 107374182400 bytes/磁盘阵列Disk /dev/sdf: 169.8 GB, 169874030592 bytes /磁盘阵列l 创建PV（假如共享存储是sdf）# pvcreate -f /dev/sdfl 创建VGrootstandbyha # vgcreate -s 128 datavg /dev/sdf Volume group datavg successfully created命令说明：vgcreate -s pesize vg_name /dev/diskname (pesize最大值为256，且必须为2的多少次幂)l 查看VG状态rootstandbyha # vgdisplay -v /dev/datavg Using volume group(s) on command line Finding volume group datavg - Volume group - VG Name datavg System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur L