Oracle 12cR1 12.1.0.1 2-node RAC on CentOS 6.4 on VMware Workstation 9 – Part 8.docx

Oracle 12cR1 12.1.0.1 2-node RAC on CentOS 6.4 on VMware Workstation 9 PartVIIIPosted on July 10, 2013 by oraclejedi Posted in Linux, Oracle, RAC, VMware Tagged 12.1.0.1, 12c, 12cR1, 12cR1 RAC, centos, CentOS 6.4, install, red hat, Red Hat 6.4, step by step, VMWare Workstation Leave a comment Time Required: 30 minutesClass Materials: Oracle 11gR2 Grid Infrastructure softwareNext we are going to perform some final steps before we can launch the Oracle Grid Infrastructure install.The Grid Infrastructure will provide the cluster software that allows the RAC nodes to communicate, as well as the ASM software to manage the shared disks.To begin, download the zip files from the Oracle software download website and unzip on Ruggero. Make sure you are logged in as the oracle user so that oracle owns the unzipped files.There are two zip files for 12cR1 Linux 64-bit called linuxamd64_12c_grid_1of2.zip and linuxamd64_12c_grid_2of2.zip.Unzip these files into a common folder. You will be left with a directory called grid.First we are going to set up secure password sharing between Ruggero and Magda. This will allow the oracle account on one machine to access resources on the other machine, without being prompted for a password.Change into the grid/sshsetup directory and we will find a script called sshUserSetup.shWe are going to use the following command to set up password sharing:./sshUserSetup.sh -user oracle -hosts ruggero magda -noPromptPassphrase -confirm -advancedThis command will not only allow the oracle account on Ruggero to access Magda, but also allow the oracle account on Magda to access Ruggero.You will be prompted for the oracle password four times during execution. I have included the full output so you can see the progress.Launch the script as follows:oracleruggero sshsetup$ ./sshUserSetup.sh -user oracle -hosts ruggero magda -noPromptPassphrase -confirm -advancedThe output of this script is also logged into /tmp/sshUserSetup_2013-07-08-22-45-18.logHosts are ruggero magdauser is oraclePlatform:- Linux Checking if the remote hosts are reachablePING ruggero (10.10.1.110) 56(84) bytes of data.64 bytes from ruggero (10.10.1.110): icmp_seq=1 ttl=64 time=0.032 ms64 bytes from ruggero (10.10.1.110): icmp_seq=2 ttl=64 time=0.041 ms64 bytes from ruggero (10.10.1.110): icmp_seq=3 ttl=64 time=0.041 ms64 bytes from ruggero (10.10.1.110): icmp_seq=4 ttl=64 time=0.044 ms64 bytes from ruggero (10.10.1.110): icmp_seq=5 ttl=64 time=0.044 ms- ruggero ping statistics -5 packets transmitted, 5 received, 0% packet loss, time 3999msrtt min/avg/max/mdev = 0.032/0.040/0.044/0.007 msPING magda (10.10.1.120) 56(84) bytes of data.64 bytes from magda (10.10.1.120): icmp_seq=1 ttl=64 time=0.486 ms64 bytes from magda (10.10.1.120): icmp_seq=2 ttl=64 time=0.590 ms64 bytes from magda (10.10.1.120): icmp_seq=3 ttl=64 time=0.449 ms64 bytes from magda (10.10.1.120): icmp_seq=4 ttl=64 time=0.435 ms64 bytes from magda (10.10.1.120): icmp_seq=5 ttl=64 time=0.486 ms- magda ping statistics -5 packets transmitted, 5 received, 0% packet loss, time 4004msrtt min/avg/max/mdev = 0.435/0.489/0.590/0.056 msRemote host reachability check succeeded.The following hosts are reachable: ruggero magda.The following hosts are not reachable: .All hosts are reachable. Proceeding further.firsthost ruggeronumhosts 2The script will setup SSH connectivity from the host ruggero.hadesnet to allthe remote hosts. After the script is executed, the user can use SSH to runcommands on the remote hosts or copy files between this host ruggero.hadesnetand the remote hosts without being prompted for passwords or confirmations.NOTE 1:As part of the setup procedure, this script will use ssh and scp to copyfiles between the local host and the remote hosts. Since the script does notstore passwords, you may be prompted for the passwords during the execution ofthe script whenever ssh or scp is invoked.NOTE 2:AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORYAND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESEdirectories.Do you want to continue and let the script make the above mentioned changes (yes/no)?Confirmation provided on the command lineThe user chose yesUser chose to skip passphrase related questions.Creating .ssh directory on local host, if not present alreadyCreating authorized_keys file on local hostChanging permissions on authorized_keys to 644 on local hostCreating known_hosts file on local hostChanging permissions on known_hosts to 644 on local hostCreating config file on local hostIf a config file exists already at /home/oracle/.ssh/config, it would be backed up to /home/oracle/.ssh/config.backup.Removing old private/public keys on local hostRunning SSH keygen on local host with empty passphraseGenerating public/private rsa key pair.Your identification has been saved in /home/oracle/.ssh/id_rsa.Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.The key fingerprint is:9b:18:cb:fc:b4:cd:98:9b:e8:18:6e:e4:f8:53:fa:9b oracleruggero.hadesnetThe keys randomart image is:+- RSA 1024-+| | | | | . S | .o.+ o | +.o= + | .=o = B | o+E.B.o |+-+Creating .ssh directory and setting permissions on remote host ruggeroTHE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.The script would create oracle/.ssh/config file on remote host ruggero. If a config file exists already at oracle/.ssh/config, it would be backed up to oracle/.ssh/config.backup.The user may be prompted for a password here since the script would be running SSH on host ruggero.Warning: Permanently added ruggero,10.10.1.110 (RSA) to the list of known hosts.oracleruggeros password: Done with creating .ssh directory and setting permissions on remote host ruggero.Creating .ssh directory and setting permissions on remote host magdaTHE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.The script would create oracle/.ssh/config file on remote host magda. If a config file exists already at oracle/.ssh/config, it would be backed up to oracle/.ssh/config.backup.The user may be prompted for a password here since the script would be running SSH on host magda.Warning: Permanently added magda,10.10.1.120 (RSA) to the list of known hosts.oraclemagdas password: Done with creating .ssh directory and setting permissions on remote host magda.Copying local host public key to the remote host ruggeroThe user may be prompted for a password or passphrase here since the script would be using SCP for host ruggero.oracleruggeros password: Done copying local host public key to the remote host ruggeroCopying local host public key to the remote host magdaThe user may be prompted for a password or passphrase here since the script would be using SCP for host magda.oraclemagdas password: Done copying local host public key to the remote host magdaCreating keys on remote host ruggero if they do not exist already. This is required to setup SSH on host ruggero.Creating keys on remote host magda if they do not exist already. This is required to setup SSH on host magda.Generating public/private rsa key pair.Your identification has been saved in .ssh/id_rsa.Your public key has been saved in .ssh/id_rsa.pub.The key fingerprint is:92:73:f2:da:38:c5:e6:41:c2:c2:f3:eb:68:5a:90:fa oraclemagda.hadesnetThe keys randomart image is:+- RSA 1024-+| | | . . | .+ o. | o +=+S | . . .*= |. . =. | . .o+. | Eo.+. |+-+Updating authorized_keys file on remote host ruggeroUpdating known_hosts file on remote host ruggeroUpdating authorized_keys file on remote host magdaUpdating known_hosts file on remote host magdacat: /home/oracle/.ssh/known_hosts.tmp: No such file or directorycat: /home/oracle/.ssh/authorized_keys.tmp: No such file or directorySSH setup is complete.-Verifying SSH setup=The script will now run the date command on the remote nodes using sshto verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FORPASSWORDS. If you see any output other than date or are prompted for thepassword, ssh is not setup correctly and you will need to resolve theissue and set up ssh again.The possible causes for failure could be:1. The server settings in /etc/ssh/sshd_config file do not allow sshfor user oracle.2. The server may have disabled public key based authentication.3. The client public key on the server may be outdated.4. oracle or oracle/.ssh on the remote host may not be owned by oracle.5. User may not have passed -shared option for shared remote users ormay be passing the -shared option for non-shared remote users.6. If there is output in addition to the date, but no password is asked,it may be a security alert shown as part of company policy. Append theadditional text to the /sysman/prov/resources/ignoreMessages.txt file.-ruggero:-Running /usr/bin/ssh -x -l oracle ruggero date to verify SSH connectivity has been setup from local host to ruggero.IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.Mon Jul 8 22:45:52 PDT 2013-magda:-Running /usr/bin/ssh -x -l oracle magda date to verify SSH connectivity has been setup from local host to magda.IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.Mon Jul 8 22:45:53 PDT 2013-Verifying SSH connectivity has been setup from ruggero to ruggeroIF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.Mon Jul 8 22:45:53 PDT 2013-Verifying SSH connectivity has been setup from ruggero to magdaIF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.Mon Jul 8 22:45:53 PDT 2013-Verification from complete-SSH verification complete.Next we are going to load a missing RPM. The cvuqdisk-1.0.9-1.rpm is located in the grid install directory under the RPM sub-directory.Log in as root and load the missing RPM.rootruggero # cd /media/ORASOL/linux_x86/grid/rpmrootruggero rpm# rpm -ivh cvuqdisk-1.0.9-1.rpmPreparing. # 100%Using default group oinstall to install package 1:cvuqdisk # 100%rootruggero rpm#Remember this RPM must be loaded on the Magda as well.Next we are going to fix a problem with runcluvfy.sh on Red Hat and Centos 6.2 and above systems. The runcluvfy.sh script is used to ensure your cluster is ready for the grid install, but if you run this on a Red Hat or CentOS system on 6.2 or greater, you get the following error:ERROR: Reference data is not available for verifying prerequisites on this operating system distributionVerification cannot proceedThis is due to bug 15973656 which states that runcluvfy.sh cannot properly detect the operating system. MOS provides a zip file attached to the bug report called clupack.zip. This zip file contains several files to patch the tool, but for Oracle 12cR1 we only need one, a dummy RPM file called redhat-release-6Server-1.noarch.rpm.If you want to use runcluvfy.sh, and I strongly recommend that you do, you will need to access MOS for bug 15973656 and download the attached zip file.Then load the RPM:rootruggero clufix# rpm -ivh redhat-release-6Server-1.noarch.rpmPreparing. # 100% 1:redhat-release We can now run the runcluvfy.sh tool to test if our cluster is ready for the install. Log into Ruggero as the oracle user and navigate back to the grid install directory you created when you unzipped the 12c grid files.In the grid directory you should see a file called runcluvfy.sh. The command we are going to use is as follows./runcluvfy.sh stage -pre crsinst -n ruggero,magdaThe above command will check the cluster against nodes ruggero and magda. If you followed all the steps carefully, you should get the following output:oracleruggero grid$ ./runcluvfy.sh stage -pre crsinst -n ruggero,magdaPerforming pre-checks for cluster services setup Checking node reachability.Node reachability check passed from node ruggeroChecking user equivalence.User equivalence check passed for user oracleChecking node connectivity.Checking hosts config file.Verification of the hosts config file successfulNode connectivity passed for subnet 192.168.0.0 with node(s) ruggero,magdaTCP connectivity check passed for subnet 192.168.0.0Node connectivity passed for subnet 10.10.1.0 with node(s) ruggero,magdaTCP connectivity check passed for subnet 10.10.1.0Node connectivity passed for subnet 10.10.2.0 with node(s) ruggero,magdaTCP connectivity check passed for subnet 10.10.2.0Interfaces found on subnet 192.168.0.0 that are likely candidates for VIP are:ruggero eth0:192.168.0.4magda eth0:192.168.0.3Interfaces found on subnet 10.10.1.0 that are likely candidates for a private interconnect are:ruggero eth1:10.10.1.110magda eth5:10.10.1.120Interfaces found on subnet 10.10.2.0 that are likely candidates for a private interconnect are:ruggero eth2:10.10.2.110magda eth4:10.10.2.120WARNING: Could not find a suitable set of interfaces with the same name for the private interconnectChecking subnet mask consistency.Subnet mask consistency check passed for subnet 192.168.0.0.Subnet mask consistency check passed for subnet 10.10.1.0.Subnet mask consistency check passed for subnet 10.10.2.0.Subnet mask consistency check passed.Node connectivity check passedChecking multicast communication.Checking subnet 192.168.0.0 for multicast communication with multicast group 224.0.0.251.Check of subnet 192.168.0.0 for multicast communication with multicast group 224.0.0.251 passed.Check of multicast communication passed.Checking ASMLib configuration.Check for ASMLib configuration passed.Total memory check passedAvailable memory check passedSwap space check passedFree disk space check passed for ruggero:/usr,ruggero:/var,ruggero:/etc,ruggero:/sbin,ruggero:/tmpFree disk space check passed for magda:/usr,magda:/var,magda:/etc,magda:/sbin,magda:/tmpCheck for multiple users with UID value 501 passed User existence check passed for oracleGroup existence check passed for oinstallGroup existence check passed for dbaMembership check for user oracle in group oinstall as Primary passedMembership check for user oracle in group dba passedRun level check passedHard limits check passed for maximum open file descriptorsSoft limits check passed for maximum open file descriptorsHard limits check passed for maximum user processesSoft limits check passed for maximum user processesSystem architecture check passedKernel version check passedKernel parameter check passed for semmslKernel parameter check passed for semmnsKernel parameter check passed for semopmKernel parameter check passed for semmniKernel parameter check passed for shmmaxKernel parameter check passed for shmmniKernel parameter check passed for shmallKernel parameter check passed for file-maxKernel parameter check passed for ip_local_port_rangeKernel parameter check passed for rmem_defaultKernel parameter check passed for rmem_maxKernel parameter check passed for wmem_defaultKernel parameter check passed for wmem_maxKernel parameter check passed for aio-max-nrPackage existence check passed for binutilsPackage existence check passed for compat-libcap1Package existence check passed for compat-libstdc+-33(x86_64)Package existence check passed for libgcc(x86_64)Package existence check passed for libstdc+(x86_64)Package existence check passed for libstdc+-devel(x86_64)Package existence check passed for sysstatPackage existence check passed for gccPackage existence check passed for gcc-c+Package existence check passed for kshPackage existence check passed for makePackage existence check passed for glibc(x86_64)Package existence check passed for glibc-devel(x86_