Oracle 10g RAC On Linux Using VMware Server.doc

Oracle 10g RAC On Linux Using VMware ServerThis article describes the installation of Oracle 10g release 2 () RAC on Linux (CentOS 4) using VMware Server with no additional shared disk devices. Introduction Download Software VMware Server Installation Virtual Machine Setup Guest Operating System Installation Oracle Installation Prerequisites Install VMware Client Tools Create Shared Disks Clone the Virtual Machine Install the Clusterware Software Install the Database Software and Create an ASM Instance Create a Database using the DBCA TNS Configuration Check the Status of the RAC IntroductionOne of the biggest obstacles preventing people from setting up test RAC environments is the requirement for shared storage. In a production environment, shared storage is often provided by a SAN or high-end NAS device, but both of these options are very expensive when all you want to do is get some experience installing and using RAC. A cheaper alternative is to use a FireWire disk enclosure to allow two machines to access the same disk(s), but that still costs money and requires two servers. A third option is to use VMware Server to fake the shared storage.Using VMware Server you can run multiple Virtual Machines (VMs) on a single server, allowing you to run both RAC nodes on a single machine. In additon, it allows you to set up shared virtual disks, overcoming the obstacle of expensive shared storage.Before you launch into this installation, here are a few things to consider. The finished system includes the host operating system, two guest operating systems, two sets of Oracle Clusterware, two ASM instances and two Database instances all on a single server. As you can imagine, this requires a significant amount of disk space, CPU and memory. I tried this installation on a 3.4G Pentium 4 with 2G of memory and it failed abysmally. When I used a dual 3.0G Xeon server with 4G of memory it worked fine, but it wasnt exactly fast. This procedure provides a bare bones installation to get the RAC working. There is no redundancy in the Clusterware installation or the ASM installation. To add this, simply create double the amount of shared disks and select the Normal redundancy option when it is offered. Of course, this will take more disk space. During the virtual disk creation, I always choose not to preallocate the disk space. This makes virtual disk access slower during the installation, but saves on wasted disk space. This is not, and should not be considered, a production-ready system. Its simply to allow you to get used to installing and using RAC. Download SoftwareDownload the following software. CentOS 4.x VMware Server Oracle 10g () CRS and DB software VMware Server InstallationFor this article, I used CentOS 4.3 as both the host and guest operating systems. Regardless of the host OS, the setup of the virtual machines should be similar.First, install the VMware Server software. On Linux you do this with the following command as the root user.# rpm -Uvh VMware-server-*.rpmPreparing. # 100% 1:VMware-server # 100%#Then finish the configuration by running the vmware-config.pl script as the root user. Most of the questions can be answered with the default response by pressing the return key. The output below shows my responses to the questions.# vmware-config.plMaking sure services for VMware Server are stopped.Stopping VMware services: Virtual machine monitor OK You must read and accept the End User License Agreement to continue.Press enter to display it.VMWARE, INC.SOFTWARE BETA TEST AGREEMENT* Editied out license agreement *Do you accept? (yes/no) yesThank you.Configuring fallback GTK+ 2.4 libraries.In which directory do you want to install the mime type icons?/usr/share/iconsWhat directory contains your desktop menu entry files? These files have a.desktop file extension. /usr/share/applicationsIn which directory do you want to install the applications icon?/usr/share/pixmapsTrying to find a suitable vmmon module for your running kernel.The module bld-2.6.9-5.EL-i686smp-RHEL4 loads perfectly in the running kernel.Do you want networking for your virtual machines? (yes/no/help) yesConfiguring a bridged network for vmnet0.The following bridged networks have been defined:. vmnet0 is bridged to eth0All your ethernet interfaces are already bridged.Do you want to be able to use NAT networking in your virtual machines? (yes/no)yesConfiguring a NAT network for vmnet8.Do you want this program to probe for an unused private subnet? (yes/no/help)yesProbing for an unused private subnet (this can take some time).The subnet / appears to be unused.The following NAT networks have been defined:. vmnet8 is a NAT network on private subnet .Do you wish to configure another NAT network? (yes/no) noDo you want to be able to use host-only networking in your virtual machines?yes noTrying to find a suitable vmnet module for your running kernel.The module bld-2.6.9-5.EL-i686smp-RHEL4 loads perfectly in the running kernel.Please specify a port for remote console connections to use 902Stopping xinetd: OK Starting xinetd: OK Configuring the VMware VmPerl Scripting API.Building the VMware VmPerl Scripting API.Using compiler /usr/bin/gcc. Use environment variable CC to override.The installation of the VMware VmPerl Scripting API succeeded.Do you want this program to set up permissions for your registered virtualmachines? This will be done by setting new permissions on all files found inthe /etc/vmware/vm-list file. no yesGenerating SSL Server CertificateIn which directory do you want to keep your virtual machine files?/var/lib/vmware/Virtual Machines /u01/VMDo you want to enter a serial number now? (yes/no/help) no yesPlease enter your 20-character serial number.Type XXXXX-XXXXX-XXXXX-XXXXX or Enter to cancel: ENTER-YOUR-SERIAL-NUMBERStarting VMware services: Virtual machine monitor OK Virtual ethernet OK Bridged networking on /dev/vmnet0 OK Host-only networking on /dev/vmnet8 (background) OK NAT service on /dev/vmnet8 OK Starting VMware virtual machines. OK The configuration of VMware Server e.x.p build-22874 for Linux for this runningkernel completed successfully.#The VMware Server Console is started by issuing the command vmware at the command prompt, or by selecting it from the System Tools menu.On the Connect to Host dialog, accept the Local host option by clicking the Connect button.You are then presented with the main VMware Server Console screen.The VMware Server is now installed and ready to use.Virtual Machine SetupNow we must define the two virtual RAC nodes. We can save time by defining one VM, then cloning it when it is installed.Click the Create a new virtual machine button to start the New Virtual Machine Wizard. Click the Next button onthe welcome page.Select the Custom virtual machine configuration and click the Next button.Select the Linux guest operating system option, and set the version to Red Hat Enterprise Linux 4, then click the Next button.Enter the name RAC1 and the location should default to /u01/VM/RAC1, then click the Next button.Select the required number of processors and click the Next button.Uncheck the Make this virtual machine private checkbox and click the Next button.Select the amount of memory to associate with the virtual machine. Remember, you are going to need two instances, so dont associate too much, but you are going to need approximately 1 Gig (1024 Meg) to compete the installation successfully.Accept the Use bridged networking option by clicking the Next button.Accept the LSI Logic option by clicking the Next button.Select the Create a new virtual disk option and click the Next button.Accept the SCSI option by clicking the Next button. Its a virtual disk, so you can still use this option even if your physical disk is IDE or SATA.Set the disk size to 10.0 GB and uncheck the Allocate all disk space now option. The latter will make disk access slower, but will save you wasting disk space.Accept RAC1.vmdk as the disk file name and complete the VM creation by clicking the Finish button.On the VMware Server Console screen, click the Edit virtual machine settings button.On the Virtual Machine Settings screen, highlight the Floppy 1 drive and click the - Remove button.Click the + Add button and select a hardware type of Ethernet Adapter, then click the Next button.Accept the Bridged option by clicking the Finish button.Click on the Options tab, highlight the Startup/Shutdown setting and select the Dont power on virtual machine in the On host startup option. Finish by clicking the OK button.The virtual machine is not configured so we can start the guest operating system installation.Guest Operating System InstallationPlace the first CentOS 4 disk in the CD drive and start the virtual machine by clicking the Power on this virtual machine button. The right pane of the VMware Server Console should display a boot loader, then the CentOS installation screen.Continue through the CentOS 4 installation as you would for a normal server. A general pictorial guide to the installation can be found here. More specifically, it should be a server installation with a minimum of 2G swap, secure Linux disabled and the following package groups installed: X Window System GNOME Desktop Environment Editors Graphical Internet Server Configuration Tools FTP Server Development Tools Legacy Software Development Administration Tools System Tools To be consistent with the rest of the article, the following information should be set during the installation: hostname: rac1.localdomain IP Address eth0: 01 (public address) IP Address eth1: 01 (private address) You are free to change the IP addresses to suit your network, but remember to stay consistent with those adjustments throughout the rest of the article.Once the basic installation is complete, install the following packages whilst logged in as the root user.# From CentOS4 Disk 1cd /media/cdrecorder/CentOS/RPMSrpm -Uvh setarch-1*rpm -Uvh compat-libstdc+-33-3*rpm -Uvh make-3*rpm -Uvh glibc-2*cd /eject# From CentOS4 Disk 2cd /media/cdrecorder/CentOS/RPMSrpm -Uvh openmotif-2*rpm -Uvh compat-db-4*rpm -Uvh gcc-3*cd /eject# From CentOS4 Disk 3cd /media/cdrecorder/CentOS/RPMSrpm -Uvh libaio-0*rpm -Uvh rsh-*rpm -Uvh compat-gcc-32-3*rpm -Uvh compat-gcc-32-c+-3*rpm -Uvh openmotif21*cd /ejectOracle Installation PrerequisitesPerform the following steps whilst logged into the RAC1 virtual machine as the root user.The /etc/hosts file must contain the following information. localhost.localdomain localhost# Public01 rac1.localdomain rac102 rac2.localdomain rac2#Private01 rac1-priv.localdomain rac1-priv02 rac2-priv.localdomain rac2-priv#Virtual11 rac1-vip.localdomain rac1-vip12 rac2-vip.localdomain rac2-vipAdd the following lines to the /etc/sysctl.conf file.kernel.shmall = 2097152kernel.shmmax = 2147483648kernel.shmmni = 4096# semaphores: semmsl, semmns, semopm, semmnikernel.sem = 250 32000 100 128fs.file-max = 65536net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default=262144net.core.rmem_max=262144net.core.wmem_default=262144net.core.wmem_max=262144Run the following command to change the current kernel parameters./sbin/sysctl -pAdd the following lines to the /etc/security/limits.conf file.* soft nproc 2047* hard nproc 16384* soft nofile 1024* hard nofile 65536Add the following line to the /etc/pam.d/login file, if it does not already exist.session required /lib/security/pam_limits.soDisable secure linux by editing the /etc/selinux/config file, making sure the SELINUX flag is set as follows.SELINUX=disabledAlternatively, this alteration can be done using the GUI tool (Applications System Settings Security Level). Click on the SELinux tab and disable the feature.Create the new groups and users.groupadd oinstallgroupadd dbagroupadd operuseradd -g oinstall -G dba oraclepasswd oracleCreate the directories in which the Oracle software will be installed.mkdir -p /u01/app/oracle/product/10.2.0/crsmkdir -p /u01/app/oracle/product/10.2.0/db_1mkdir -p /u01/oradatachown -R oracle.oinstall /u01During the installation, both RSH and RSH-Server were installed. Enable remote shell and rlogin by doing the following.chkconfig rsh onchkconfig rlogin onservice xinetd reloadCreate the /etc/hosts.equiv file as the root user.touch /etc/hosts.equivchmod 600 /etc/hosts.equivchown root:root /etc/hosts.equivEdit the /etc/hosts.equiv file to include all the RAC nodes: +rac1 oracle+rac2 oracle+rac1-priv oracle+rac2-priv oracleLogin as the oracle user and add the following lines at the end of the .bash_profile file.# Oracle SettingsTMP=/tmp; export TMPTMPDIR=$TMP; export TMPDIRORACLE_BASE=/u01/app/oracle; export ORACLE_BASEORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1; export ORACLE_HOMEORACLE_SID=RAC1; export ORACLE_SIDORACLE_TERM=xterm; export ORACLE_TERMPATH=/usr/sbin:$PATH; export PATHPATH=$ORACLE_HOME/bin:$PATH; export PATHLD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATHCLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATHif $USER = oracle ; then if $SHELL = /bin/ksh ; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fifiInstall VMware Client ToolsLogin as the root user on the RAC1 virtual machine, then select the VM Install VMware Tools. option from the main VMware Server Console menu.This should mount a virtual CD containing the VMware Tools software. Double-click on the CD icon labelled VMware Tools to open the CD. Right-click on the .rpm package and select the Open with Install Packages menu option.Click the Continue button on the Completed System Preparation screen and wait for the installation to complete.Once the package is loaded, the CD should unmount automatically. You must then run the vmware-config-tools.pl script as the root user. The following listing is an example of the output you should expect.# vmware-config-tools.plStopping VMware Tools services in the virtual machine: Guest operating system daemon: OK Trying to find a suitable vmhgfs module for your running kernel.The module bld-2.6.9-11.EL-i686up-RHEL4 loads perfectly in the running kernel.pcnet32 30409 0Unloading pcnet32 moduleTrying to find a suitable vmxnet module for your running kernel.The module bld-2.6.9-11.EL-i686up-RHEL4 loads perfectly in the running kernel.Detected X.org version 6.8.Please choose one of the following display sizes (1 - 13):1 640x4802 800x6003 1024x7684 1152x8645 1280x8006 1152x9007 1280x10248 1376x10329 1400x105010 1680x105011 1600x120012 1920x120013 2364x1773Please enter a number between 1 and 13:12 3X Window System Version 6.8.2Release Date: 9 February 2005X Protocol Version 11, Revision 0, Release 6.8.2Build Operating System: Linux 2.6.9-11.EL i686 ELFCurrent Operating System: Linux rac1.localdomain 2.6.9-22.EL #1 Sat Oct 8 17:48:27 CDT 2005 i686Build Date: 07 October 2005Build Host: x8664-build.home.local Before reporting problems, check http:/wiki.X.Org to make sure that you have the latest version.Module Loader presentOS Kernel: Linux version 2.6.9-22.EL (buildcentoslouisa.home.local) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2) #1 Sat Oct 8 17:48:27 CDT 2005 PMarkers: (-) probed, (*) from config file, (=) default setting, (+) from command line, (!) notice, (II) informational, (WW) warning, (EE) error, (NI) not implemented, (?) unknown.(+) Log file: /tmp/vmware-config0/XF86ConfigLog.3674, Time: Thu Apr 13 21:17:37 2006(+) Using config file: /tmp/vmware-config0/XF86Config.3674X is running fine with the new config file.(WW) VMWARE(0): Failed to set up write-combining range (0xf0000000,0x1000000)Starting VMware Tools services in the virtual machine: Switching to guest configuration: OK Guest filesystem driver: OK Guest vmxnet fast network device: OK DMA setup: OK Guest operating system daemon: OK The configuration of VMware Tools e.x.p build-22874 for