HACMP Oracle Rac安装设置手册

1、HACMP Oracle Rac安装设置手册版本 v1.0二零一二年十月北京华胜天成科技股份有限公司文档控制更改记录版本创建/修改时间编制/修改者 文件/修改内容审批人V1.02012.10审阅姓名职位发布姓名职位目录 TOC o h z u HYPERLINK l _Toc225161693 目录 PAGEREF _Toc225161693 h 3 HYPERLINK l _Toc225161694 第1章 HACMP v5.4安装配置 PAGEREF _Toc225161694 h 4 HYPERLINK l _Toc225161695 1.1、 IP规划 PAGEREF _Toc2251

2、61695 h 4 HYPERLINK l _Toc225161696 1.2、 安装HACMP软件 PAGEREF _Toc225161696 h 4 HYPERLINK l _Toc225161697 1.2.1、 操作系统版本 PAGEREF _Toc225161697 h 4 HYPERLINK l _Toc225161698 1.2.2、 安装依赖的软件包 PAGEREF _Toc225161698 h 5 HYPERLINK l _Toc225161699 1.2.3、 安装HACMP v5.4 PAGEREF _Toc225161699 h 6 HYPERLINK l _Toc2

3、25161700 1.3、 HACMP配置规划 PAGEREF _Toc225161700 h 8 HYPERLINK l _Toc225161701 1.3.1、 配置IP和网络 PAGEREF _Toc225161701 h 8 HYPERLINK l _Toc225161702 1.3.2、 编辑/etc/hosts文件 PAGEREF _Toc225161702 h 9 HYPERLINK l _Toc225161703 1.3.3、 创建共享卷组和文件系统 PAGEREF _Toc225161703 h 9 HYPERLINK l _Toc225161704 1.4、 HACMP S

4、tandard配置 PAGEREF _Toc225161704 h 10 HYPERLINK l _Toc225161705 1.4.1、 添加 cluster和节点 PAGEREF _Toc225161705 h 11 HYPERLINK l _Toc225161706 1.4.2、 创建并配置资源组 PAGEREF _Toc225161706 h 11 HYPERLINK l _Toc225161707 1.4.3、 同步HACMP的配置 PAGEREF _Toc225161707 h 13 HYPERLINK l _Toc225161708 1.5、 HACMP启动 PAGEREF _T

5、oc225161708 h 13 HYPERLINK l _Toc225161709 1.5.1、 启动HACMP PAGEREF _Toc225161709 h 13 HYPERLINK l _Toc225161710 第2章 Oracle RAC安装配置 PAGEREF _Toc225161710 h 13 HYPERLINK l _Toc225161711 2.1、 规划 PAGEREF _Toc225161711 h 13 HYPERLINK l _Toc225161712 2.2、 安装检查 PAGEREF _Toc225161712 h 15 HYPERLINK l _Toc225

6、161713 2.3、 安装前配置 PAGEREF _Toc225161713 h 17 HYPERLINK l _Toc225161714 2.4、 安装CRS PAGEREF _Toc225161714 h 20 HYPERLINK l _Toc225161715 2.5、 安装数据库 PAGEREF _Toc225161715 h 41 HYPERLINK l _Toc225161716 2.6、 创建数据库 PAGEREF _Toc225161716 h 47 HYPERLINK l _Toc225161717 2.7、 数据测试 PAGEREF _Toc225161717 h 59H

7、ACMP v5.4安装配置IP规划NodeInterface NameTypeIP AddressRegistered Inibm03ibm03Public8Hosts fileibm03ibm03-vipVirtual10Hosts fileibm03ibm03-privPrivate8Hosts fileibm04ibm04Public9Hosts fileibm04ibm04-vipVirtual11Hosts fileibm04ibm04-privPrivate9Hosts file安装HACMP软件完成HACMP安装配置规划后，可以开始安装HACMP软件，安装版本为HACMP v5.

8、4。操作系统版本# oslevel -r5300-06安装依赖的软件包检查系统的软件安装包，请参照IBM红皮书中High Availability Cluster Multi-Processing for AIX 5L Installation Guide。这里我们建议安装软件束APP-DEV和SERVER，其中包括的红皮书要求的大部分软件包。安装rsct软件包：rsct.basicpat.basicpat.clientsrsct.opt.storagerm（可选）安装其它软件包：bos.clvm（使用并行资源组时需要，RAC下必须安装此包）bos.data#smitty install_la

9、testInstall SoftwareType or select values in entry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /dev/cd0SOFTWARE to install _all_latestPREVIEW only? (install operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMA

10、TICALLY install requisite software? yesEXTEND file systems if space needed? yesOVERWRITE same or newer versions? noVERIFY install and check file sizes? noInclude corresponding LANGUAGE filesets? yesDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE

11、agreements? no在“SOFTWARE to install”一栏中选择要安装的软件包，“ACCEPT new license agreements”中接受许可协议，即完成安装软件包。或参考以下脚本命令：#安装软件包/usr/lib/instl/sm_inst installp_cmd -a -Q -d /dev/cd0 -f bos.clvm ALL I:bos.clvm _all_filesets,bos.data ALL I:bos.data _all_filesets,rsct.basic ALL I:rsct.basic _all_filesets,pat.basic AL

12、L I:pat.basic _all_filesets,pat.clients ALL I:pat.clients _all_filesets,rsct.opt.storagerm ALL I:rsct.opt.storagerm _all_filesets -c -N -g -X -G -Y #安装HACMP v5.4除了haview ，netwiew ( Tivoli),的 包以外，所有的HACMP的软件包都要安装，rsct.opt.storagerm为可选安装包，对于基本包在安装光盘或Expansion光盘上可以查找到。#smitty install_latestInstall Soft

13、wareType or select values in entry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /dev/cd0SOFTWARE to install _all_latestPREVIEW only? (install operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMATICALLY install

14、requisite software? yesEXTEND file systems if space needed? yesOVERWRITE same or newer versions? noVERIFY install and check file sizes? noInclude corresponding LANGUAGE filesets? yesDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE agreements? no在“

15、SOFTWARE to install”一栏中选择要安装的软件包，“ACCEPT new license agreements”中接受许可协议，即完成安装软件包，安装过程大约在1030分钟完成。或参考以下脚本命令：#安装HACMP/usr/lib/instl/sm_inst installp_cmd -a -Q -d /dev/cd0 -f cluster.adt.es ALL I:cluster.adt.es _all_filesets,cluster.doc.en_US.es ALL I:cluster.doc.en_US.es _all_filesets,cluster.es ALL I

16、:cluster.es _all_filesets,cluster.es.cfs ALL I:cluster.es.cfs _all_filesets,cluster.es.cspoc ALL I:cluster.es.cspoc _all_filesets,cluster.es.nfs ALL I:cluster.es.nfs _all_filesets,cluster.es.plugins ALL I:cluster.es.plugins _all_filesets,cluster.es.worksheets ALL I:cluster.es.worksheets _all_fileset

17、s,cluster.license ALL I:cluster.license _all_filesets,cluster.man.en_US.es ALL I:cluster.man.en_US.es _all_filesets -c -N -g -X -G -Y系统提示成功后，完成HACMP补丁升级，将HACMP补丁上传至系统目录下，我们假设上传目录为/tmp/hav54，本地目录为d:hav54，上传命令如下：Windows：“开始”-“运行”，输入“cmd”，根据以下提示完成补丁上传：Microsoft Windows XP 版本 5.1.2600(C) 版权所有 1985-2001

18、Microsoft Corp.C:Documents and SettingsAdministratorftp remote-host-ip-address输入用户名和密码（此处需要给用户设置密码，如果为空，AIX拒绝登录），成功登录后：ftpcd /tmp/hav54ftplcd d:hav54ftppromptftpbinftpmput *.bff上传完成后退出ftp程序ftpquit升级HACMP v5.4补丁#smitty update_all Update Installed Software to Latest Level (Update All)Type or select va

19、lues in entry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /tmp/hav54SOFTWARE to update _update_allPREVIEW only? (update operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMATICALLY install requisite software? y

20、esEXTEND file systems if space needed? yesVERIFY install and check file sizes? noDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE agreements? no回车确认以上命令，即可完成升级工作。注意：如果安装程序提示“没有找到相关的fileset”等类似的提示，可在/tmp/hav54目录下执行命令“#inutoc .”，重新生成软件包安装列表。重新启动AIX安

21、装完成HACMP后，为避免各种问题，要求将AIX系统重新启动，至此我们完成了所有的安装工作，可以开始配置HACMP双机集群了。#shutdown FrHACMP配置规划配置IP和网络AIX网卡IP地址配置：#smiitty mktcpip Available Network InterfacesMove cursor to desired item and press Enter.en0 08-08 Standard Ethernet Network Interfaceen1 08-09 Standard Ethernet Network Interface en2 0A-08 Standar

22、d Ethernet Network Interface en3 0J-08 Standard Ethernet Network Interface选择想要配置IP地址的网卡，选择“en0”，回车确认，进入下一步配置：Minimum Configuration & StartupTo Delete existing configuration data, please use Further Configuration menusType or select values in entry fields.Press Enter AFTER making all desired changes.

23、Entry FieldsHOSTNAME ora01Internet ADDRESS (dotted decimal) 2Network MASK (dotted decimal) Network INTERFACE en0NAMESERVERInternet ADDRESS (dotted decimal) DOMAIN Name Default GatewayAddress (dotted decimal or symbolic name) Cost 0Do Active Dead Gateway Detection? noYour CABLE Type N/ASTART TCP/IP d

24、aemons Now no使用同样的方法配置完成网卡的IP地址。编辑/etc/hosts文件对于两个节点ibm03和ibm04要配置相同的/etc/hosts文件，使用vi配置hosts文件，建议首先对原有的hosts文件进行备份，配置如下：#cp /etc/hosts /etc/hosts.bak#vi /etc/hosts8 ibm039 ibm048 ibm03-priv9 ibm04-priv10 ibm03-vip11 ibm04-vip创建共享卷组和文件系统创建卷组#smitty _mkbvgAdd a Big Volume GroupType or select values i

25、n entry fields.Press Enter AFTER making all desired changes.Entry FieldsVOLUME GROUP name vgdataPhysical partition SIZE in megabytesPHYSICAL VOLUME names hdisk2FORCE the creation of volume group? yesActivate volume group AUTOMATICALLY noat system restart?Volume group MAJOR NUMBER 100Create VG Concur

26、rent Capable? no我们在建立HACMP共享卷组时，要注意的参数 “Activate volume group AUTOMATICALLY at system restart? ”该项，设置为“no”，是使用HACMP的方式将卷组varyon，而不是系统启动时由操作系统varyon。Major NUMBER是指该设备（orvg）的主设备号，此处设置为系统唯一即可，可通过命令查询可用的major number：# lvlstmajor43.59,62.或可通过ls l /dev/命令查看当前所有设备的主设备号，第五列即是。创建逻辑卷和文件系统在ibm03节点上按照以下操作：#vary

27、offvg vgdata需将vgdata varyoff下来，在ora02节点上才能正确的导入。在ibm04节点上按照一下操作：#lspv hdisk0 00c2ef1fa38df28a rootvg activehdisk1 00c2ef1f38b81fac rootvg activehdisk2 00c2ecef3846c20e None 查看hdisk2是否为物理卷，否则需要使用命令将其属性更改：#chdev l hdisk2 a pv=yes导入卷组和文件系统的定义#smitty importvgImport a Volume GroupType or select values in

28、 entry fields.Press Enter AFTER making all desired changes.Entry FieldsVOLUME GROUP name vgdata* PHYSICAL VOLUME name hdisk2Volume group MAJOR NUMBER 100#varyonvg vgdata#varyoffvg vgdataHACMP Standard配置HACMP配置可采用Standard和Extended两种方式，其中Standard配置可以配置Cluter名称，节点，基本网络，应用服务器，各种资源和资源组等基本的内容，如果要配置永久IP，串口

29、心跳，对资源组进行详细的配置等工作则需要在Extended中完成，可以说Standard菜单是Extended菜单的一个子集，可以在Extended菜单中完成HACMP的所有配置工作。下面我们以Standard菜单完成HACMP的基本配置工作：添加cluster和节点配置cluster资源创建并配置资源组同步HACMP的配置HACMP配置菜单如下：#smitty hacmp HACMP for AIXMove cursor to desired item and press Enter. Initialization and Standard Configuration Extended Co

30、nfiguration System Management (C-SPOC) Problem Determination Tools选择“Initialization and Standard Configuration”，如下菜单：Initialization and Standard ConfigurationMove cursor to desired item and press Enter.Configuration AssistantsConfigure an HACMP Cluster and NodesConfigure Resources to Make Highly Ava

31、ilableConfigure HACMP Resource GroupsVerify and Synchronize HACMP ConfigurationDisplay HACMP ConfigurationHACMP Cluster Test Tool添加 cluster和节点#smitty hacmp-Initialization and Standard ConfigurationConfigure an HACMP Cluster and NodesConfigure Nodes to an HACMP Cluster (standard)Type or select values

32、 in entry fields.Press Enter AFTER making all desired changes.Entry Fields* Cluster Name ora_clusterNew Nodes (via selected communication paths) ibm03 ibm04Currently Configured Node(s)创建并配置资源组#smitty hacmp- Initialization and Standard ConfigurationConfigure HACMP Resource Groups Configure HACMP Reso

33、urce GroupsMove cursor to desired item and press Enter. Add a Resource Group Change/Show a Resource Group Remove a Resource Group Change/Show Resources for a Resource Group (standard)Add a Resource GroupType or select values in entry fields.Press Enter AFTER making all desired changes.Entry Fields*

34、Resource Group Name ora_rg* Participating Nodes (Default Node Priority) ibm03 ibm04Startup Policy Online On All AvailabFallover Policy Bring Offline (On ErrFallback Policy Never Fallback添加资源至资源组中#smitty hacmp- Initialization and Standard ConfigurationConfigure HACMP Resource Groups Configure HACMP R

35、esource GroupsMove cursor to desired item and press Enter. Add a Resource Group Change/Show a Resource Group Remove a Resource Group Change/Show Resources for a Resource Group (standard)Change/Show All Resources and Attributes for a Resource GroupType or select values in entry fields.Press Enter AFT

36、ER making all desired changes.Entry FieldsResource Group Name ora_rgParticipating Nodes (Default Node Priority) ibm03 ibm04Startup Policy Online On Home Node OnlyFallover Policy Fallover To Next Priority Node In The ListFallback Policy Fallback To Higher Priority Node In The ListConcurrent Volume Gr

37、oups vgdata +Use forced varyon of volume groups, if necessary false +Automatically Import Volume Groups false +Application Servers +Tape Resources + 至此，我们完成了资源和资源组的关联。同步HACMP的配置#smitty hacmp- Initialization and Standard ConfigurationVerify and Synchronize HACMP Configuration回车确认该操作，这样HACMP的配置已经同步到两个

38、节点中了，也就是ibm03和ibm04拥有同样的配置了。HACMP启动启动HACMP在节点ibm03上执行smitty clstart，回车确认即可启动HACMP* Start now, on system restart or both now + Start Cluster Services on these nodes ibm03 + BROADCAST message at startup? true + Startup Cluster Information Daemon? false + Reacquire resources after forced down ? false +

39、Oracle RAC安装配置规划数据库配置规划ORACLE_BASE/dataCRS_HOME/data/product /10.2.0/crsORACLE_HOME /data/product /10.2.0/dbDB_NAMEorclORACLE_SIDorcl1，orcl2TNS_ADMIN$ORACLE_HOME/network/adminOracle管理帐户的口令 sys Systemoracleoracle数据库存放位置裸设备是否使用归档方式运行数据库Yes备份方式说明RMAN/TSM裸设备规划设备名称数据库设施存储空间/dev/rorcl_ocrOcr信息存放500Mb/dev/

40、rorcl_vdiskvdisk信息存放500Mb/dev/rorcl_spfilespfile信息存放100Mb/dev/rorcl_srvconfigsrvconfig信息存放100Mb/dev/rorcl_systemSystem表空间10G/dev/rorcl_sysauxSysaux表空间10G/dev/rorcl_temp1系统临时表空间20G/dev/rorcl_temp2系统临时表空间20G/dev/rorcl_undo1撤销表空间20G/dev/rorcl_undo2撤销表空间20G/dev/rorcl_undo3撤销表空间20G/dev/rorcl_undo4撤销表空间20

41、G/dev/rorcl_usersUser表空间10G/dev/rorcl_toolsTools表空间10G/dev/rorcl_ctl1控制文件1G/dev/rorcl_ctl2控制文件1G/dev/rorcl_ctl3控制文件1G/dev/rorcl_log1a日志文件500Mb/dev/rorcl_log1b日志文件500Mb/dev/rorcl_log2a日志文件500Mb/dev/rorcl_log2b日志文件500Mb/dev/rorcl_log3a日志文件500Mb/dev/rorcl_log3b日志文件500Mb/dev/rorcl_log4a日志文件500Mb/dev/ror

42、cl_log4b日志文件500Mb/dev/rorcl_data1用户数据表空间20G/dev/rorcl_data2用户数据表空间20G/dev/rorcl_data3用户数据表空间20G/dev/rorcl_data4用户数据表空间20G安装检查操作系统版本：# oslevel -r5300-06物理内存：# lsattr -El mem0goodsize 7744 Amount of usable physical memory in Mbytes Falsesize 7744 Total amount of physical memory in Mbytes False交换区：# ls

43、ps -aPage Space Physical Volume Volume Group Size %Used Active Auto Typehd6 hdisk0 rootvg 512MB 3 yes yes lv系统空间：# df -gFilesystem GB blocks Free %Used Iused %Iused Mounted on/dev/hd4 0.50 0.44 13% 4788 5% /dev/hd2 5.00 3.10 38% 41240 6% /usr/dev/hd9var 1.00 0.95 6% 747 1% /var/dev/hd3 1.00 0.65 36%

44、 1082 1% /tmp/dev/fwdump 0.38 0.37 1% 4 1% /var/adm/ras/platform/dev/hd1 0.50 0.50 1% 71 1% /home/proc - - - - - /proc/dev/hd10opt 2.00 1.86 7% 3461 1% /opt/dev/fslv00 15.00 4.42 71% 29243 3% /data/dev/lv00 0.12 0.12 4% 18 1% /var/adm/csd文件集：# lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.

45、libperfstat bos.perf.perfstat ctools rsct.basic.rte pat.clients.rte xlC.aix50.rte xlC.rte Fileset Level State Description Path: /usr/lib/objrepos bos.adt.base COMMITTED Base Application Development Toolkit bos.adt.lib COMMITTED Base Application Development Libraries bos.adt.libm COMMITTED Base Appli

46、cation Development Math Library bos.perf.libperfstat COMMITTED Performance Statistics Library Interface bos.perf.perfstat COMMITTED Performance Statistics Interface ctools COMMITTED Proc Filesystem Tools rsct.basic.rte COMMITTED RSCT Basic Function pat.clients.rte COMMITTED RSCT Event Management Cli

47、ent Function xlC.aix50.rte COMMITTED XL C/C+ Runtime for AIX 5.2 xlC.rte COMMITTED XL C/C+ Runtime Path: /etc/objrepos bos.perf.libperfstat COMMITTED Performance Statistics Library Interface bos.perf.perfstat COMMITTED Performance Statistics Interface rsct.basic.rte COMMITTED RSCT Basic Function安装前配

48、置为确保oracle rac安装成功，系统管理员root用户需创建相应的系统用户、系统组，以便安装管理oracle软件。Root用户还要修改相关系统参数，打破默认值对实际应用的限制。调试rac节点之间的高级功能：rcp/rsh/rlogin，以确定两节点之间可以通信正常，可以在rac环境下进行相关同步操作和心跳监控。创建dba组，id为500# smitty mkgroup创建oracle用户，id为601# smitty mkuser修改oracle用户密码# passwd oracle在oracle用户下创建.rhosts文件，内容为“+”。# su oracle$ vi .rhosts+

49、修改oracle用户限制# smitty chuserSoft FILE size -1Soft CPU time -1Soft DATA segment -1Soft STACK size -1Soft CORE file size -1修改系统配置参数，增加系统进程数上限# smitty chgsysMaximum number of PROCESSES allowed per user 2048修改网络参数no -p -o udp_sendspace=65536no -p -o udp_recvspace=65536no -p -o tcp_sendspace=65536no -p -o

50、 tcp_recvspace=65536no -p -o rfc1323=1no -p -o sb_max=1310720no -r -o ipqmaxlen=512创建裸设备mklv -y rac_system_raw_500m -T O -w n -s n -r n vgdata 8依次创建以下裸设备：/dev/rorcl_ocr/dev/rorcl_vdisk/dev/rorcl_spfile/dev/rorcl_srvconfig/dev/rorcl_system/dev/rorcl_sysaux/dev/rorcl_temp1/dev/rorcl_temp2/dev/rorcl_un

51、do1/dev/rorcl_undo2/dev/rorcl_undo3/dev/rorcl_undo4/dev/rorcl_users/dev/rorcl_tools/dev/rorcl_ctl1/dev/rorcl_ctl2/dev/rorcl_ctl3/dev/rorcl_log1a/dev/rorcl_log1b/dev/rorcl_log2a/dev/rorcl_log2b/dev/rorcl_log3a/dev/rorcl_log3b/dev/rorcl_log4a/dev/rorcl_log4b/dev/rorcl_data1/dev/rorcl_data2/dev/rorcl_d

52、ata3/dev/rorcl_data4更改裸设备权限# cd /dev# chown oracle:dba *orcl*修改oracle用户环境变量# vi /home/oracle/.profileumask 022export ORACLE_BASE=/dataexport ORACLE_HOME=$ORACLE_BASE/product/10.2.0/dbexport ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crsexport LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/lib32export O

53、RACLE_SID=orclexport PATH=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:$PATH安装CRS在ibm03节点上进入CRS安装盘，执行runInstaller启动CRS的安装；按提示在每个安装节点上以root用户执行rootpre.sh，该脚本在CRS安装盘上的rootpre目录下；# ./rootpre.sh./rootpre.sh output will be logged in /tmp/rootpre.out_06-04-13.12:33:04Saving the original files in /etc/ora_save_06-04

54、-13.12:33:04Copying new kernel extension to /etcLoading the kernel extension from /etc Oracle Kernel Extension Loader for AIX Copyright (c) 1998,1999 Oracle Corporation Successfully loaded /etc/pw-syscall.64bit_kernel with kmid: 0 x441e300 Successfully configured /etc/pw-syscall.64bit_kernel with km

55、id: 0 x441e300The kernel extension was successfuly loaded.Configuring Asynchronous I/OAsynchronous I/O is already definedConfiguring POSIX Asynchronous I/OChecking if group services should be configuredGroup hagsuser does not exist.Creating required group for group services: hagsuserPlease add your

56、Oracle userid to the group: hagsuserConfiguring HACMP group services socket for possible use by Oracle.The group or permissions of the group services socket have changed.Please stop and restart HACMP before trying to use Oracle.在两个节点都执行root.sh脚本：ibm03# ./root.shWARNING: directory /oracle/product/10.

57、2.0 is not owned by rootWARNING: directory /oracle/product is not owned by rootChecking to see if Oracle CRS stack is already configuredChecking to see if any 9i GSD is upSetting the permissions on OCR backup directorySetting up NS directoriesOracle Cluster Registry configuration upgraded successful

58、lyWARNING: directory /oracle/product/10.2.0 is not owned by rootWARNING: directory /oracle/product is not owned by rootSuccessfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 1: ibm03 ibm03-priv ibm03node 2: ibm04 ibm04-priv ibm04Creating OC

59、R keys for user root, privgrp system.Operation successful.Now formatting voting device: /dev/rora_voteFormat of 1 voting devices complete.Startup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.CSS is active on these nodes. ibm03CSS is inactive on these nodes. ibm04Local node checking complete.Run root.sh on remaining nodes to start CRS daemons.ibm04 # ./root.shWARNING: directory /oracle/product/10.2.0 is not owned by