在vmware vsphere esxi上安装oracle RAC

1、Oracle Databases on VMwareRAC 安装手册Contents1.Introduction52.Infrastructure Environment62.1Hardware62.2Software Environment62.3VMware Virtual Machines62.4Virtual Disk Layout72.5Oracle RAC Database Environment72.6Network Configuration72.7Physical Architecture102.8Logical Architecture113.Overview of Dep

2、loyment Steps124.Configure NTP Client on All ESX Hosts135.Prepare a Virtual Machine for the First RAC Node155.1Create a Virtual Machine155.2Add Two NICs155.3Add CRS and Voting Disk165.4Install the Guest Operating System195.5Install VMware Tools215.6Timekeeping with the Linux Guest Operating System29

3、6.Prepare Subsequent Virtual Machines/RAC Nodes306.1Clone the First Virtual Machine/RAC Node306.2Configure CSR and Voting Disks326.3Format the Added Disks Using fdisk347.Install Oracle Grid Infrastructure358.Verify the Grid Infrastructure Installation379.Add DATA and REDO Disks to Virtual Machine RA

4、C Nodes3910.Create DATA and REDO ASM Disk Groups as Grid User4311.Install and Create the Oracle Database 11g R2 RAC4412.References451. 介绍本文通过一个案例简要介绍 Oracle Real Application Clusters (RAC) 在 VMware vSphere® 平台上的安装过程, 帮助 Oracle 数据库管理员在VMware 平台上规划和实施Oracle RAC环境. 在VMware环境下,数据库管理员通常需要和VMware的管理员

5、以及存储厂商的管理员合作,以顺利部署一个Oracle RAC环境.在虚拟化平台上部署Oracle RAC在许多方面与纯物理环境上的部署很类似,比如:· 在虚拟机创建完成之后,安装操作系统和Oracle RAC软件的过程与物理环境下基本一致· 存储的规划,包括用于数据存放的LUN数量的规划和多路径的设计与物理环境也基本一致.差别在于我们可以充分依靠虚拟化软件, 简化许多工作, 从而加快安装进度,减少错误的发生,比如:· 利用事先准备好的模板, 多次安装同版本的系统和数据库,同时加强标准化· 利用快照技术, 很方便的进行测试和回退· 利用虚拟机复制

6、技术,简化多个RAC节点的系统安装、打补丁、配置工作由于安装过程的细节与软件版本（尤其是Linux版本和Oracle版本）有较强的关联性，本文只能尽量把安装步骤描述清楚.读者在自己进行安装的时候，还需要参考相应版本的官方安装文档.先将本次安装中用到的主要软件和版本罗列如下：· VMware ESXi 5.1 · VMware vCenter Server 5.1 · vSphere Client 5.1 · Oracle Database 11g . · Oracle Enterprise Linux x86_64 6.3.2. 软硬件环境2.

7、1 硬件组件数量配置ESXi Hosts2Cisco UCS B4404x 10-core CPUs, 256GB RAM, and 4 NICsStorage1NetApp F3240 双路控制器Ethernet Switches2Cisco FI6248FC Switches1 Cisco Nexus 7000Host Based Adapters4每个刀片两个，与网卡共享流量2.2 软件环境组件版本作用vSphere5.1Hypervisor hosting all virtual machinesVMware vCenter5.1管理 vSphereOracle Enterprise

8、Linux x86_646.3Guest operating system for virtual machines running Oracle Database serverOracle Database 11g (with Oracle RAC and Oracle Grid Infrastructure) Enterprise EditionMicrosoft Windows Server2008 R2用于 vCenter Server2.3 虚拟机配置虚拟机用途数量配置Oracle RAC nodes220 vCPU, 120GB RAM,Oracle Enterpr

9、ise Linux x86_64 6.3vCenter12 vCPUs, 8GB RAM, Windows Server 2008 R22.4 共享存储配置Datastore NameVirtual Disk on ESXLinux路径虚拟路径驱动类型VMDK Size (GB)用途VMDATASTOREVMDK Hard disk 1/dev/sdaSCSI 0:0LSI Logic150Oracle Enterprise Linux 5.5 OS and Oracle binariesCRS1VMDK Hard disk 2Shared disk/dev/sdb1SCSI 1:0Parav

10、irtual20CRS and voting diskREDO1VMDK Hard disk 3Shared disk/dev/sdc1SCSI 1:1Paravirtual900REDO and ArchieveDATA01VMDK Hard disk 4Shared disk/dev/sdd1SCSI 2:0Paravirtual1000RAC database DATADATA02VMDK Hard disk 5Shared disk/dev/sde1SCSI 2:1Paravirtual1000RAC database DATADATA03VMDK Hard disk 6Shared

11、disk/dev/sdf1SCSI 3:0Paravirtual1000RAC database DATADATA04VMDK Hard disk 7Shared disk/dev/sdg1SCSI 3:1Paravirtual1000RAC database DATA2.5 Oracle RAC 数据库环境Node NameASM InstanceRAC InstanceDatabaseASM StorageRAC1+ASM1RAC1RACASM file systemRAC2+ASM2RAC2RACASM file system2.6 网络配置按照Oracle RAC的实施惯例，每个RAC

12、节点都提供一个Public网口用于承载业务，并在其上捆绑虚拟IP地址 (VIP) 用于HA切换. 各RAC节点之间还需要一个私有网络进行节点之间的通信。在物理环境中，通常需要考虑链路的冗余以消除单点鼓掌。而在虚拟化环境中，冗余问题由底层虚拟化软件处理，所以对于每个RAC节点，只是简单地需要两个网口就可以工作，大大简化了配置和安装工作.我们这次测试的网络配置如下三:· UCS底层软件配置出三个物理网口（均为10Gb） vmnic0 和 vmnic1用于Oracle Public and Oracle Private网络, vmnic2用于vSphere® vMotion

13、74;工作.· 两个分布式虚拟交换机 Oracle Public (VM Net) and Oracle Private (Oracle HB).· 每个RAC节点配置3个IP地址 以节点1 (RAC1) 为例:o One public static IP address (eth0): 162.1o One private static IP address (eth1):o One virtual static IP address(VIP): 162.1（由CRS软件配置和管理）还有三个虚拟IP用于节点间负载均衡（由CRS配置和管理）物理网口 vmnic0 连接到分布

14、式交换机 VM Net,vmnic1连接到分布式交换机Oracle HB， vMotion 使用标准交换机vSwitch3. 这样区分概念上比较清晰，也可以只建一个分布式网络，用不同的端口组区分数据网络和心跳网。Figure 1. Network ConfigurationOracle RAC Private2.7 2.8 逻辑结构图 Figure 2. Logical Architecture of Two-Node Oracle RAC on vSphere3. 部署流程概览部署流程大体按照下面的流程图进行.Figure 3. Deployment Steps for Four-Node

15、Oracle RAC on vSphere with VMFS4. 为第一个RAC节点准备虚拟机环境本节安装步骤简介如下:· 使用vSphere Client创建第一个虚拟机。添加公网和心跳网卡，挑选适当的LUN创建仲裁盘CRS （VMDK模式）。· 安装Redhat Linux操作系统，安装VMware tools，按照Oracle RAC要求安装相应补丁.· 在Linux中配置NTP 客户端.4.1 创建第一个虚拟机1. 使用vSphere Client登录vCenter.2. 选中一台主机,在Getting Started 页面单击 Create a new

16、 virtual machine.3. In the Create New Virtual Machine wizard, select Typical for the Configuration, and click Next.4. On the Name and Location page, enter RAC1 for the Name, and click Next.5. On the Datastore page, select VMDATASTORE, and click Next.6. On the Guest Operating System page, select Linu

17、x from the Guest Operating System list. For the Version, select Oracle Linux 6 (64-bit) from the drop-down menu. Click Next.7. On the Create a Disk page, set the Virtual Disk Size to 20GB, and click Next.8. On the Ready to Complete page, select Edit the virtual machine settings before completion. Cl

18、ick Continue.9. On the Virtual Machine Properties page for the RAC1 virtual machine, select 8 CPUs and 128GB Memory.10. Click Finish.4.2 增加两个虚拟网卡现在需要增加两块网卡. 一块用于公共数据网络, 另一块用于心跳网络. 添加这两块网卡的过程基本相同,差别仅在于添加公网网卡时选择标签: VM Net 而添加心跳网卡时选择 Oracle HB.添加公网网卡的步骤：1. Select a virtual machine.2. Click Edit setting

19、s to display the Virtual Machine Properties pane.3. Click Add.4. Select Ethernet Adaptor and click Next to display the Add Hardware wizard.5. On the Network connection page, select VMXNET 3 from the Adapter Type drop-down menu.6. In the Network Connection section of the Network connection page, sele

20、ct Named network with specified label.7. In the Named network with specified label drop-down menu, select VM Net.8. In the Device Status section of the Network connection page, select Connect at power on.9. On the Ready to Complete page, click Finish, and then click OK.添加心跳网卡的步骤：1. Repeat steps 1 th

21、rough 6 from the preceding procedure.2. In the Named network with specified label drop-down menu, select Oracle HB.3. Continue with steps 8 and 9 from the preceding procedure.4.3 添加仲裁盘Oracle RAC需要至少一个共享的仲裁盘存放clusterware信息和节点投票信息。存储管理员需要事先为仲裁盘分配好单独的LUN，然后我们需要创建单独的SCSI控制器并设为Paravirtual驱动. SCSI 总线 shar

22、ing policy 设为 none.下面描述创建共享存储的步骤. 为虚拟机创建 20GB datastore：1. In vSphere Client, select a virtual machine.2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.3. Click Add to display the Add Hardware wizard.4. In the Add Hardware wizard

23、select Hard Disk for the Device Type, and click Next.5. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 20GB, select the Support clustering features such as Fault Tolerance check box, and select CRS1 to specify a datastore. Also, for vSphere 5, i

24、n the Disk Provisioning section select Thick Provision Eager Zeroed. Click Next.6. On the Compatibility Mode page, click Next.7. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In the Mode section, select Independent. Click Next.8. On the Ready to Complete page,

25、click Finish.9. Click OK.Add configuration parameters for each Oracle RAC virtual machine as described in Disabling simultaneous write protection provided by VMFS using the multi-writer flag ().手工修改虚拟机参数1. In vSphere Client, select a virtual machine.2. Right-click the name of the virtual machine and

26、 select Edit Settings to display the Virtual Machine Properties pane.3. Click the Options tab, click Advanced, and click General.4. Click Configuration Parameters.5. On the Configuration Parameters page, add rows and entries for the shared disks in the virtual machine's configuration parameters

27、as shown in the following screenshot.6. Repeat steps 1 through 5 for each of the remaining virtual machines.You can view the virtual machine properties for the RAC1 virtual machine in vCenter as in the following screenshot.web client 显示如下图：4.4 安装操作系统将OEL6.4安装介质ISO文件上传到datastore，然后进行安装。Note: This doc

28、ument is specific to Linux guest operating system. Oracle RAC and above versions can be deployed and supported on the list of guest operation systems using VMware multi-writer flag ().在虚拟机上安装Linux操作系统1. In vSphere Client, select a virtual machine.2. Click Edit settings.3. In the Hardware ta

29、b of the Virtual Machine Properties pane, select CD/DVD Drive 1.4. In the Device Status section of this pane, select the Connect at power on check box.5. In the Device Type section of this pane, select Datastore ISO File and enter the path to your Linux ISO image file.6. In the Virtual Device Node s

30、ection of this pane, select CD/DVD Drive 1 from the drop-down menu.7. Click OK.8. Power on the virtual machine.9. Install Linux with the following settings.Custom partitioning scheme:/dev/sda1 ext3 150GB mounted on /Installed packages use the default package selection, plus the following:· kern

31、el-uek.x86_64· kmod-oracleasm· (download from .)··10. Use below script to install remaining packages:yum install binutils compat-libcap1 compat-libstdc+-33 compat-libstdc+-33.i686 gcc.x86_64 gcc-c+.x86_64 glibc.x86_64 glibc.i686 glibc-devel.x86_64 glibc-devel.i686 unixODBC.i686 u

32、nixODBC.x86_64 unixODBC-devel.x86_64 unixODBC-devel.i686 libgcc.x86_64 libgcc.i686 libstdc+.x86_64 libstdc+.i686 libstdc+-devel.x86_64 libstdc+-devel.i686 compat-libstdc+-33.x86_64 compat-libstdc+-33.i686 libaio.x86_64 libaio.i686 libaio-devel.x86_64 libaio-devel.i686 sysstat.x86_64 ksh libXtst-deve

33、l.x86_64 libXtst.i686 compat-db.x86_64 xorg-x11-utils.x86_64 xorg-x11-xinit.x86_64 make11. View the installed packages with this command:run command “rpm q <package name>12. Confirm that the following packages have been installed:· binutils-.2-5.11.el6 (x86_64)· compat-libca

34、p1-1.10-1 (x86_64)· compat-libstdc+-33-3.2.3-69.el6 (x86_64)· compat-libstdc· gcc-4.4.4-13.el6 (x86_64)· gcc-c+-4.4.4-13.el6 (x86_64)· glibc-2.12-1.7.el6 (i686)· glibc-2.12-1.7.el6 (x86_64)· glibc-devel-2.12-1.7.el6 (x86_64)·· ksh· libgcc-4.4.4-13.el

35、6 (i686)· libgcc-4.4.4-13.el6 (x86_64)· libstdc+-4.4.4-13.el6 (x86_64)· libstdc· libstdc+-devel-4.4.4-13.el6 (x86_64)· libstdc· libaio-0.3.107-10.el6 (x86_64)·· libaio-devel-0.3.107-10.el6 (x86_64)···· sysstat-9.0.4-11.el6 (x86_64)· un

36、ixODBC-2.2.14-11.el6 (x86_64) or later· unixODBC-2.2.14-11.el6.i686 or later· unixODBC-devel-2.2.14-11.el6 (x86_64) or later· unixODBC-devel-2.2.14-11.el6.i686 or later13. 最后安装下面两个包:· oracle-rdbms-server-11gR2-preinstall (从YUM安装，这个包会配置Oracle RAC所需系统参数和环境变量)· cvuqdisk (Oracle

37、安装介质中提供这个包，用于安装前检查)14. 检查配置文件/etc/sysctl.confkernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 6815744 (512 x processes for example 6815744 for 13312 processes)net.ipv4.ip_local_port_range = 9000 65500  = 262144  = 4194304  = 262144  = 1048576 

38、kernel.shmall = physical RAM size / pagesize For most systems, this will be the value 2097152. See Note: 301830.1 for more information. eg:kernel.shmall=4160749568(62GB)kernel.shnmax=   34359738368(32GB)kernel.shmmax = RAM times 0.5 (or higher at customer's discretion - seeNo

39、te:567506.1) /etc/profile ulimit u 16384 ulimit n 6553515. 依照oracle传统，创建clusterware主用户 “grid”:rootracnode1 # groupadd -g 1000 oinstall rootracnode1 # groupadd -g 1200 asmadminrootracnode1 # groupadd -g 1201 asmdba rootracnode1 # groupadd -g 1202 asmoper rootracnode1 # useradd -m -u 1100 -g oins

40、tall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid rootracnode1 # id grid uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall), 1200(asmadmin), 1201(asmdba),1202(asmoper) 设置 grid 帐户的口令：rootracnode1 # passwd grid Changing password for

41、 user grid. New UNIX password:xxxxxxxxxxx Retype new UNIX password:xxxxxxxxxxx passwd: all authentication tokens updated successfully. 为grid用户创建初始化脚本(/home/grid/.bash_profile):# -# .bash_profile# -# OS User: grid# Application: Oracle Grid Infrastructure# Version: Oracle 11g release 2# -# Get the ali

42、ases and functionsif -f /.bashrc ; then . /.bashrcfialias ls="ls -FA"# -# ORACLE_SID# -# Specifies the Oracle system identifier (SID)# for the Automatic Storage Management (ASM)instance# running on this node.# Each RAC node must have a unique ORACLE_SID.# (i.e. +ASM1, +ASM2,.)# -ORACLE_SID

43、=+ASM1; export ORACLE_SID# -# JAVA_HOME# -# Specifies the directory of the Java SDK and Runtime# Environment.# -JAVA_HOME=/usr/local/java; export JAVA_HOME# -# ORACLE_BASE# -# Specifies the base of the Oracle directory structure# for Optimal Flexible Architecture (OFA) compliant# installations. The

44、Oracle base directory for the# grid installation owner is the location where# diagnostic and administrative logs, and other logs# associated with Oracle ASM and Oracle Clusterware# are stored.# -ORACLE_BASE=/u01/app/grid; export ORACLE_BASE# -# ORACLE_HOME# -# Specifies the directory containing the

45、Oracle# Grid Infrastructure software. For grid# infrastructure for a cluster installations, the Grid# home must not be placed under one of the Oracle base# directories, or under Oracle home directories of# Oracle Database installation owners, or in the home# directory of an installation owner. Durin

46、g # installation, ownership of the path to the Grid # home is changed to root. This change causes # permission errors for other installations.# -ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME# -# ORACLE_PATH# -# Specifies the search path for files used by Oracle# applications such as SQL*Plus.

47、 If the full path to# the file is not specified, or if the file is not# in the current directory, the Oracle application# uses ORACLE_PATH to locate the file.# This variable is used by SQL*Plus, Forms and Menu.# -ORACLE_PATH=/u01/app/oracle/common/oracle/sql; export ORACLE_PATH# -# SQLPATH# -# Speci

48、fies the directory or list of directories that# SQL*Plus searches for a login.sql file.# -# SQLPATH=/u01/app/common/oracle/sql; export SQLPATH# -# ORACLE_TERM# -# Defines a terminal definition. If not set, it# defaults to the value of your TERM environment# variable. Used by all character mode produ

49、cts. # -ORACLE_TERM=xterm; export ORACLE_TERM# -# NLS_DATE_FORMAT# -# Specifies the default date format to use with the# TO_CHAR and TO_DATE functions. The default value of# this parameter is determined by NLS_TERRITORY. The# value of this parameter can be any valid date# format mask, and the value must be surrounded by # double quotation marks. For example:# # NLS_DATE_FORMAT = "MM/DD/YYYY"# # -NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS" export NLS_DATE_FORMAT# -# TNS_ADMIN# -# Specifies the directory containing the Oracle Net# Servi