RHEL6.3安装Oracle11gR2(ASM+iSCSI+FreeNAS).doc

RHEL6.3安装Oracle11gR2（ASM+iSCSI+FreeNAS）从Oracle 11g开始，ASM(Automatic Storage Management)作为Grid Infrastructure的一部分，要想在Oracle数据库使用ASM，需要安装GridInfrastructure。而且从RHEL 6起，Oracle不再提供ASMlib的核心驱动，如果想使用ASM必须使用Oracle的Unbreakable Enterprise Kernel(UEK)代替RedHat的kernel。本文将描述如何在RHEL 6.3上使用UEK安装Oracle 11g并使用ASM存储数据库文件，并且使用FreeNAS作为数据库存储系统。 RHEL 6.3和FreeNAS都在VmwareWorkstation 8.0.4中安装。FreeNAS是一套基于FreeBSD操作系统核心的开放源代码的网络存储设备（Network-AttachedStorage，简称NAS）服务器系统，支持众多服务，用户访问权限管理，提供网页设置接口。RHEL6FreeNAS操作系统RHEL 6 update 3 X86_64FreeNAS-8.3.0-RELEASE-p1 X86_64主机名oraeIP地址/子网掩码/24/24 安装和配置FreeNAS RHEL中安装UEK核心 配置RHEL的iscsi 配置Grid Infrastructure和Oracle 11g 数据库安装环境 创建ASM Disk Volumes 安装Grid Infrastructure 创建ASM Disk Groups 安装Oracle11g数据库 创建数据库实例一、安装和配置FreeNAS1.1新建虚拟机如何新建这里不具体描述，要注意的是新建时Guest Operating System选择other-FreeBSD 64-bit，新建完后如下图所示FreeNAS有5块磁盘，每块磁盘有如下作用：Hard Disk (5GB)-安装FreeNAS系统Hard Disk 2 (1GB)-存储CRS(ClusterReady Services)文件Hard Disk 3 (20GB)-存储数据库文件Hard Disk 4 (20GB)-存储数据库文件Hard Disk 5 (15GB)-数据库快速恢复区1.2安装FreeNASFreeNAS安装很简单，记住要把FreeNAS安装到5GB的磁盘上。1.3配置FreeNAS如果网络中有dhcp服务器，安装完后会自动获取ip地址，我在这里手工指定ip地址为/24。之后我们可以在浏览器中输入进入FreeNAS的配置页面。点击左边的树形菜单Services-iSCSI-Target GlobalConfiguration进入iSCSI的设置页面,如图图 iSCSI-1。图 iSCSI-11.3.1配置Portal地址IP Address选择刚设置的地址，其他默认，如图图 iSCSI-2。图 iSCSI-21.3.2配置Inilator点击Add initiator按钮，在弹出的对话框中直接点ok保存，如图iSCSI-3。图 iSCSI-31.3.3配置AuthorizedAccess点击Add iSCSI Authorized Access按钮，在弹出的对话框中填入如图 iSCSI-4所示内容，此处GroupId、User和Sercret的值可以任意填写，此处分别设置为1、grid和gridasm，图 iSCSI-41.3.4配置TargetGlobal Configuration如图 iSCSI-5所示，Discovery Auth Method选择CHAP,Discovery Auth Group选择1也就是1.3.3中建立的GroupId，其他值默认即可。图 iSCSI-51.3.5配置Targets点击Add Target按钮，在弹出的对话框中填写如图iSCSI-6内容。Target Name-任意填写Target Flags-选择read-writePortal Group ID-选择1Initiator Group ID-选择1.3.3中增加的Group Id值1Auth Method-选择CHAP其他默认图 iSCSI-61.3.6配置DeviceExtents点击Add Device Extent按钮，在弹出的对话框中设Extent Name并选择Disk Device，如图iSCSI-7所示。添加完四块磁盘后页面显示的内容类似图iSCSI-8。图 iSCSI-7图 iSCSI-81.3.7配置AssociatedTargets点击Add Extent to Target按钮，在弹出的对话框中Target选择1.3.5中增加的TargetName，Extents选择1.3.6中添加的Device Extents，如果iSCSI-9。添加完后所有的Extents后得到类似图iSCSI-10的结果。图 iSCSI-9图 iSCSI-101.3.8启动iSCSI服务点击页面Services，在页面中开启iSCSI，如图iSCSI-11。至此FreeNAS已安装并配置完毕。图 iSCSI-11二、安装UEK核心UEk可以从/下载安装：rootora # wget /repo/OracleLinux/OL6/UEK/latest/x86_64/kernel-uek-2.6.39-300.17.3.el6uek.x86_64.rpmrootora # wget /repo/OracleLinux/OL6/UEK/latest/x86_64/kernel-uek-firmware-2.6.39-300.17.3.el6uek.noarch.rpmrootora # wget /repo/OracleLinux/OL6/3/base/x86_64/oracleasm-support-2.1.5-1.el6.x86_64.rpmrootora # wget /otn_software/asmlib/oracleasmlib-2.0.4-1.el6.x86_64.rpmrootora Downloads# rpm -ivhkernel-uek-firmware-2.6.39-300.17.3.el6uek.noarch.rpmrootora Downloads# rpm -ivhkernel-uek-2.6.39-300.17.3.el6uek.x86_64.rpmrootora Downloads# rpm -ivhoracleasm-support-2.1.5-1.el6.x86_64.rpmrootora Downloads# rpm -ivhoracleasmlib-2.0.4-1.el6.x86_64.rpm因为安装完后要重启所以顺便设置SELINUX为disabled，否则ASM不能加载FreeNAS上的磁盘。rootora # vi /etc/sysconfig/selinuxSELINUX=disabled安装完后重启在grub的menu菜单选择新安装的2.6.39-300.17.3.el6uek.x86_64核心启动。启动完成后检查linux核心和SELINUX是否disabled。rootora # uname -aLinuxora 2.6.39-300.17.3.el6uek.x86_64 #1 SMP Wed Dec 19 06:28:03 PST 2012 x86_64x86_64 x86_64 GNU/Linuxrootora # /usr/sbin/getenforce Disabled三、配置RHEL的iSCSI3.1查看系统是否安装iscsi软件包rootora # rpm -qa | grep iscsiiscsi-initiator-utils-72-41.el6.x86_64该软件包在RHEL的安装光盘Packages目录下。3.2编辑/etc/iscsi/iscsi.con文件rootora #vi /etc/iscsi/iscsi.con保证文件中有如下内容,grid和gridasm是在1.3.3中设置的User和Sercretnode.session.auth.username= gridnode.session.auth.password= gridasmdiscovery.sendtargets.auth.username= griddiscovery.sendtargets.auth.password= gridasm3.3查找iSCSI设备rootora # iscsiadm -m discovery -t sendtargets -p:3260,1.istgt:asmrootora # /etc/init.d/iscsi restartStoppingiscsi: OK Startingiscsi: OK 运行fdsk命令可以发现有4个新磁盘设备出现，/dev/sdb、/dev/sdc、/dev/sdd、/dev/sde分别对应FreeNAS中设置的四个Disk Extents。rootora # fdisk -lDisk/dev/sdb: 1073 MB, 1073741312 bytes34heads, 61 sectors/track, 1011 cylindersUnits= cylinders of 2074 * 512 = 1061888 bytesSectorsize (logical/physical): 512 bytes / 512 bytesI/Osize (minimum/optimal): 4096 bytes / 1048576 bytesDiskidentifier: 0x00000000Disk/dev/sdc: 21.5 GB, 21474835968 bytes64heads, 32 sectors/track, 20479 cylindersUnits= cylinders of 2048 * 512 = 1048576 bytesSectorsize (logical/physical): 512 bytes / 512 bytesI/Osize (minimum/optimal): 4096 bytes / 1048576 bytesDiskidentifier: 0x00000000Disk/dev/sdd: 21.5 GB, 21474835968 bytes64heads, 32 sectors/track, 20479 cylindersUnits= cylinders of 2048 * 512 = 1048576 bytesSectorsize (logical/physical): 512 bytes / 512 bytesI/Osize (minimum/optimal): 4096 bytes / 1048576 bytesDiskidentifier: 0x00000000Disk/dev/sde: 32.2 GB, 32212254208 bytes64heads, 32 sectors/track, 30719 cylindersUnits= cylinders of 2048 * 512 = 1048576 bytesSectorsize (logical/physical): 512 bytes / 512 bytesI/Osize (minimum/optimal): 4096 bytes / 1048576 bytesDiskidentifier: 0x00000000对新找到的四块磁盘分区，每块磁盘分一个主分区。rootora # fdisk /dev/sdbDevicecontains neither a valid DOS partition table, nor Sun, SGI or OSF disklabelBuildinga new DOS disklabel with disk identifier 0x6dcc40a0.Changeswill remain in memory only, until you decide to write them.Afterthat, of course, the previous content wont be recoverable.Warning:invalid flag 0x0000 of partition table 4 will be corrected by w(rite)WARNING:DOS-compatible mode is deprecated. Its strongly recommended toswitch off the mode (command c) andchange display units tosectors (command u).Command(m for help): nCommandactione extendedp primary partition (1-4)pPartitionnumber (1-4): 1Firstcylinder (1-1011, default 1): Usingdefault value 1Lastcylinder, +cylinders or +sizeK,M,G (1-1011, default 1011): Usingdefault value 1011Command(m for help): wThepartition table has been altered!Callingioctl() to re-read partition table.Syncingdisks.rootora # fdisk /dev/sdc略rootora # fdisk /dev/sdd略rootora # fdisk /dev/sde略分区完后再次查看磁盘信息rootora# fdisk -lDisk/dev/sdb: 1073 MB, 1073741312 bytes34heads, 61 sectors/track, 1011 cylindersUnits= cylinders of 2074 * 512 = 1061888 bytesSectorsize (logical/physical): 512 bytes / 512 bytesI/Osize (minimum/optimal): 4096 bytes / 1048576 bytesDiskidentifier: 0x6dcc40a0Device Boot Start End Blocks Id System/dev/sdb1 1 1011 1048376+ 83 LinuxPartition1 does not start on physical sector boundary.Disk/dev/sdc: 21.5 GB, 21474835968 bytes64heads, 32 sectors/track, 20479 cylindersUnits= cylinders of 2048 * 512 = 1048576 bytesSectorsize (logical/physical): 512 bytes / 512 bytesI/Osize (minimum/optimal): 4096 bytes / 1048576 bytesDiskidentifier: 0x4de8bcdbDevice Boot Start End Blocks Id System/dev/sdc1 1 20479 20970480 83 LinuxDisk/dev/sde: 32.2 GB, 32212254208 bytes64heads, 32 sectors/track, 30719 cylindersUnits= cylinders of 2048 * 512 = 1048576 bytesSectorsize (logical/physical): 512 bytes / 512 bytesI/Osize (minimum/optimal): 4096 bytes / 1048576 bytesDiskidentifier: 0xfe021ea9Device Boot Start End Blocks Id System/dev/sde1 1 30719 31456240 83 LinuxDisk/dev/sdd: 21.5 GB, 21474835968 bytes64heads, 32 sectors/track, 20479 cylindersUnits= cylinders of 2048 * 512 = 1048576 bytesSectorsize (logical/physical): 512 bytes / 512 bytesI/Osize (minimum/optimal): 4096 bytes / 1048576 bytesDiskidentifier: 0x0c852d1bDevice Boot Start End Blocks Id System/dev/sdd1 1 20479 20970480 83 Linux四、配置Grid Infrastructure和Oracle 11g 数据库安装环境4.1创建所需要的组和用户grid用户用来安装GridInfrastructure，oracle用户用来安装Oracle数据库。rootora # groupadd -g 501 oinstallrootora # groupadd -g 502 dbarootora # groupadd -g 503 operrootora # groupadd -g 601 asmadminrootora # groupadd -g 602 asmdbarootora # groupadd -g 603 asmoperrootora # useradd -u 500 -g oinstall -G dba,asmdba,operoraclerootora # passwd oracleChangingpassword for user oracle.Newpassword: Retypenew password: passwd:all authentication tokens updated successfully.rootora # id oracleuid=500(oracle)gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),602(asmdba)rootora # useradd -u 600 -g oinstall -Gdba,asmadmin,asmdba,asmoper gridrootora # passwd gridChangingpassword for user grid.Newpassword: Retypenew password: passwd:all authentication tokens updated successfully.rootora # id griduid=600(grid)gid=501(oinstall)groups=501(oinstall),502(dba),601(asmadmin),602(asmdba),603(asmoper)4.2创建安装目录rootora # mkdir -p /u01/app/gridrootora # mkdir -p /u01/app/11.2.0/gridrootora # chown -R grid:oinstall /u01rootora # mkdir -p /u01/app/oraclerootora # chown oracle:oinstall /u01/app/oraclerootora # chmod -R 775 /u014.3配置软件环境设置grid用户的的.bash_profiel文件。gridora $ cat .bash_profile ORACLE_SID=+ASM ORACLE_BASE=/u01/app/gridORACLE_HOME=/u01/app/11.2.0/gridPATH=$PATH:$HOME/bin:$ORACLE_HOME/binexport ORACLE_SID ORACLE_BASE ORACLE_HOME PATHEOFgridora $source .bash_profiel设置oracle用户的的.bash_profiel文件。oracleora $ cat .bash_profile ORACLE_SID=orcl ORACLE_BASE=/u01/app/oracle ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1 PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin export ORACLE_SID ORACLE_BASE ORACLE_HOME PATH EOForacleora $ source .bash_profiel设置核心参数。rootora Packages# cat /etc/sysctl.conf fs.aio-max-nr = 1048576fs.file-max = 6815744kernel.shmall = 2097152kernel.shmmax = 4294967295kernel.shmmni = 4096kernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048576EOF设置Oracle软件的资源限制。rootora # cat /etc/security/limits.conf grid soft nproc 2047grid hard nproc 16384grid soft nofile 1024grid hard nofile 65536oracle soft nproc 2047oracle hard nproc 16384oracle soft nofile 1024oracle hard nofile 65536EOFrootora # cat /etc/profile if $USER = oracle | $USER = grid ; thenif $SHELL = /bin/ksh ; thenulimit -p 16384ulimit -n 65536elseulimit -u 16384 -n 65536fiumask 022fiEOF修改hosts文件，使里面有如下内容。rootora # vi /etc/hosts localhost.localdomain localhost ora五、创建ASM Disk Volumes5.1配置并装载ASM核心模块rootora # oracleasm configure -iConfiguringthe Oracle ASM library driver.Thiswill configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determinewhether the driver isloadedon boot and what permissions it will have. The current valueswillbe shown in brackets (). Hitting without typing ananswerwill keep that current value. Ctrl-Cwill abort.Defaultuser to own the driver interface : gridDefaultgroup to own the driver interface : asmadminStartOracle ASM library driver on boot (y/n) n: yScanfor Oracle ASM disks on boot (y/n) y: yWritingOracle ASM library driver configuration: donerootora # oracleasm initCreating/dev/oracleasm mount point: /dev/oracleasmLoadingmodule oracleasm: oracleasmMountingASMlib driver filesystem: /dev/oracleasm5.2创建ASM磁盘rootora # oracleasm createdisk CRSVOL1 /dev/sdb1Writingdisk header: doneInstantiatingdisk: donerootora # oracleasm createdisk DATAVOL1 /dev/sdc1Writingdisk header: doneInstantiatingdisk: donerootora # oracleasm createdisk DATAVOL2 /dev/sdd1Writingdisk header: doneInstantiatingdisk: donerootora # oracleasm createdisk FRAVOL1 /dev/sde1Writingdisk header: doneInstantiatingdisk: donerootora # oracleasm listdisksCRSVOL1DATAVOL1DATAVOL2FRAVOL1dbc使用oracleasm-discover查找ASM磁盘，所以先运行oracleasm-discover查看是否能找到刚创建的4个磁盘rootora # oracleasm-discoverUsingASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.soASMLibrary - Generic Linux, version 2.0.4 (KABI_V2)Discovereddisk: ORCL:CRSVOL1 2096753 blocks (1073537536 bytes), maxio 512Discovereddisk: ORCL:DATAVOL1 41940960 blocks (21473771520 bytes), maxio 512Discovereddisk: ORCL:DATAVOL2 41940960 blocks (21473771520 bytes), maxio 512Discovereddisk: ORCL:FRAVOL1 62912480 blocks (32211189760 bytes), maxio 512六、安装GridInfrastructure用grid用户安装Grid Infrastructure，安装版本为 x86_64位版，安装过程不具体描述，但是在安装过程中有几步有以下几个要注意的地方：在Step 2 of 9时选择第二个Configure Oracle Grid Infrastructurefor a Standalone Server，如图grid-1。图grid-1在Step 4 of 11时只选择一个ASM磁盘CRSVOL1，如图grid-2。图grid-2在Step 7 of 11时会弹出警告对话框，可以不用理会，直接点击Yes，如图grid-3。图grid-3在Step 9 of 12会提示找不到pdksh-XXX的包，在RHEL6中已经移出此包，只要系统中有ksh-XXX包即可，选中IgnoreAll后点击Next按钮，如图grid-4。rootora # rpm -qa | grep kshksh-20100621-16.el6.x86_64图grid-3最后按照屏幕提示以root用户执行两个脚本，再等待oracle自动完成最后的配置，Grid Infrastructure就安装完成，如图grid-4、grid-5。图grid-4图grid-5安装完后可以查看ASM实例是否启动。rootora# ps -ef | grep asmgrid 23212 1 0 17:24 ? 00:00:00 asm_pmon_+ASMgrid 23214 1 0 17:24 ? 00:00:00 asm_psp0_+ASMgrid 23246 1 0 17:24 ? 00:00:01 asm_vktm_+ASMgrid 23250 1 0 17:24 ? 00:00:00 asm_gen0_+ASMgrid 23252 1 0 17:24 ? 00:00:00 asm_diag_+ASMgrid 23254 1 0 17:24 ? 00:00:00 asm_dia0_+ASMgrid 23256 1 0 17:24 ? 00:00:00 asm_mman_+ASMgrid 23258 1 0 17:24 ? 00:00:00 asm_dbw0_+ASMg