oracle 10g RAC linux vmware workstation FREENAS共享存储.docx

RAC安装文档一 、规划软件规划软件名称软件版本备注信息系统rhel-server-5.5-x86_64系统内核2.6.18-194.el5-2.0.5-1.el5.x86_64Oracle cluster10201_clusterware_linux_x86_64Oracle database10201_database_linux_x86_64Oracle ASMoracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64oracleasmlib-2.0.4-1.el5.x86_64oracleasm-support-2.1.7-1.el5.x86_64FreeNASFreeNAS-i386-LiveCD-543.isoIp地址规划Rac1Rac2Priv10/2411/24Public/24/24Vip53/2454/24Oracle_sidOradb1Oradb2二、安装freeNAS，作为共享存储FreeNAS是一套免费的NAS服务器，它能将一部普通PC变成网络存储服务器。该软件基于FreeBSD，Samba 及PHP，支持CIFS (samba), FTP, NFS protocols, Software RAID (0,1,5) 及 web 界面的设定工具。用户可通过Windows、Macs、FTP、SSH 及网络文件系统 (NFS) 来访问存储服务器；FreeNAS可被安装于硬盘或移动介质USB Flash Disk上，所占空间不足 16MB。1、freeNAS软件安装将镜像文件放入光驱，启动选择9安装，回车选择3，安装完全系统在HDD+DATA+SWAP partition 将会创建三个分区1.OS 2.DATA 3.SWAP 选择安装介质 选择安装分区 ad0 OS分区大小默认380M 是否创建是swap分区，yesSwap分区默认大小256M 完成安装，重新启动系统启动完成后，设置ip地址：53/24设置完成重置web登录密码，选择3重置web界面密码为：freenas 帐号是：admin登录：修改语言环境-修改为简体中文：Systemgeneral: save 保存。2、添加磁盘，模拟存储添加三块磁盘做raid5 ，给rac做共享存储-OK重新启动NAS服务器，登录管理界面找到磁盘管理点击最右侧下方的+号图标，添加磁盘选项：已格式化的文件系统选择 softraid，其他默认将三块磁盘依次添加完成添加完成后-应用更改硬盘-软raid选择raid5 ，并点击右侧+号，添加Raid名称为rac，选择三块硬盘勾选“建立并初始raid”-“添加”应用更改：初始化raid，中间需要几分钟时间；完成：3、配置ISCSI配置ISCSI目标：打开后，首先设置端口，默认端口是3260“应用更改”配置ISCSI服务发起端定义“应用更改”配置ISCSI服务目标端定义-“应用更改”打开“设置”勾选右侧启动保存并重启服务4、主机连接freeNAS首先在服务器上安装安装iscsi，iscsi-initiator-utils.x86_64 0:71-0.16.el5，并开启服务，使用iscsiadm连接freeNASrootrac1 # yum install *iscsi*iscsi-initiator-utils.x86_64 0:71-0.16.el5rootrac2 # /etc/init.d/iscsi startiscsid is stopped OK iSCSI daemon: OK OK Setting up iSCSI targets: iscsiadm: No records found! OK rootrac1 # iscsiadm -m discovery -t sendtargets -p 5353:3260,1 iqn.2007-09.jp.ne.peach.istgt:disk0rootrac1 # iscsiadm -m node -T iqn.2007-09.jp.ne.peach.istgt:disk0 -p 53 -lLogging in to iface: default, target: iqn.2007-09.jp.ne.peach.istgt:disk0, portal: 53,3260Login to iface: default, target: iqn.2007-09.jp.ne.peach.istgt:disk0, portal: 53,3260: successful使用fdisk -l查看共享存储连接情况，两台机器均已连接rootrac1 # fdisk -lDisk /dev/hda: 21.4 GB, 21474836480 bytes255 heads, 63 sectors/track, 2610 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/hda1 * 1 13 104391 83 Linux/dev/hda2 14 778 6144862+ 83 Linux/dev/hda3 779 816 305235 82 Linux swap / Solaris/dev/hda4 817 2610 14410305 5 Extended/dev/hda5 817 2610 14410273+ 83 LinuxDisk /dev/sda: 42.9 GB, 42949410816 bytes64 heads, 32 sectors/track, 40959 cylindersUnits = cylinders of 2048 * 512 = 1048576 bytes查看rac节点rootrac2 # fdisk -lDisk /dev/hda: 21.4 GB, 21474836480 bytes255 heads, 63 sectors/track, 2610 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/hda1 * 1 13 104391 83 Linux/dev/hda2 14 778 6144862+ 83 Linux/dev/hda3 779 816 305235 82 Linux swap / Solaris/dev/hda4 817 2610 14410305 5 Extended/dev/hda5 817 2610 14410273+ 83 LinuxDisk /dev/sda: 42.9 GB, 42949410816 bytes64 heads, 32 sectors/track, 40959 cylindersUnits = cylinders of 2048 * 512 = 1048576 bytesDisk /dev/sda doesnt contain a valid partition table重启iscsi服务，查看存储挂载情况rootrac1 # /etc/init.d/iscsi restartLogging out of session sid: 1, target: iqn.2007-09.jp.ne.peach.istgt:disk0, portal: 53,3260Logout of sid: 1, target: iqn.2007-09.jp.ne.peach.istgt:disk0, portal: 53,3260: successfulStopping iSCSI daemon: iscsid dead but pid file exists OK iSCSI daemon: OK OK Setting up iSCSI targets: Logging in to iface: default, target: iqn.2007-09.jp.ne.peach.istgt:disk0, portal: 53,3260Login to iface: default, target: iqn.2007-09.jp.ne.peach.istgt:disk0, portal: 53,3260: successful OK 至此共享存储配置完成；共享存储划分分区，因为存储是做的raid5，存储容量大约在40G左右，所以划分三个主分区，作为ASM磁盘组使用，另外划分若干扩展分区，使用作为安装OCR和spfile；这里只需要在一个节点上做就可以了，因为存储是共享的。rootrac1 # fdisk /dev/sdaDevice contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabelBuilding a new DOS disklabel. Changes will remain in memory only,until you decide to write them. After that, of course, the previouscontent wont be recoverable.The number of cylinders for this disk is set to 40959.There is nothing wrong with that, but this is larger than 1024,and could in certain setups cause problems with:1) software that runs at boot time (e.g., old versions of LILO)2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK)Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)Command (m for help): nCommand action e extended p primary partition (1-4)pPartition number (1-4): 1First cylinder (1-40959, default 1): Using default value 1Last cylinder or +size or +sizeM or +sizeK (1-40959, default 40959): +10000mCommand (m for help): nCommand action e extended p primary partition (1-4)pPartition number (1-4): 2First cylinder (9539-40959, default 9539): Using default value 9539Last cylinder or +size or +sizeM or +sizeK (9539-40959, default 40959): +10000mCommand (m for help): nCommand action e extended p primary partition (1-4)pPartition number (1-4): 3First cylinder (19077-40959, default 19077): Using default value 19077Last cylinder or +size or +sizeM or +sizeK (19077-40959, default 40959): +10000mCommand (m for help): nCommand action e extended p primary partition (1-4)eSelected partition 4First cylinder (28615-40959, default 28615): Using default value 28615Last cylinder or +size or +sizeM or +sizeK (28615-40959, default 40959): +12000mCommand (m for help): nFirst cylinder (28615-40059, default 28615): Using default value 28615Last cylinder or +size or +sizeM or +sizeK (28615-40059, default 40059): +1024mCommand (m for help): nFirst cylinder (29593-40059, default 29593): Using default value 29593Last cylinder or +size or +sizeM or +sizeK (29593-40059, default 40059): +1024mCommand (m for help): nFirst cylinder (30571-40059, default 30571): Using default value 30571Last cylinder or +size or +sizeM or +sizeK (30571-40059, default 40059): +1024mCommand (m for help): nFirst cylinder (31549-40059, default 31549): Using default value 31549Last cylinder or +size or +sizeM or +sizeK (31549-40059, default 40059): +4000mCommand (m for help): nFirst cylinder (35365-40059, default 35365): Using default value 35365Last cylinder or +size or +sizeM or +sizeK (35365-40059, default 40059): +4000mCommand (m for help): wqThe partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks.分区划分完成后，检查分区结果，两个节点分别做检查。rootrac1 # fdisk -lDisk /dev/hda: 22.5 GB, 22548578304 bytes255 heads, 63 sectors/track, 2741 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/hda1 * 1 25 200781 83 Linux/dev/hda2 26 286 2096482+ 82 Linux swap / Solaris/dev/hda3 287 2741 19719787+ 83 LinuxDisk /dev/sda: 42.9 GB, 42949410816 bytes64 heads, 32 sectors/track, 40959 cylindersUnits = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System/dev/sda1 1 9538 9766896 83 Linux/dev/sda2 9539 19076 9766912 83 Linux/dev/sda3 19077 28614 9766912 83 Linux/dev/sda4 28615 40059 11719680 5 Extended/dev/sda5 28615 29592 1001456 83 Linux/dev/sda6 29593 30570 1001456 83 Linux/dev/sda7 30571 31548 1001456 83 Linux/dev/sda8 31549 35364 3907568 83 Linux/dev/sda9 35365 39180 3907568 83 Linuxrootrac2 # fdisk -lDisk /dev/hda: 22.5 GB, 22548578304 bytes255 heads, 63 sectors/track, 2741 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System/dev/hda1 * 1 25 200781 83 Linux/dev/hda2 26 286 2096482+ 82 Linux swap / Solaris/dev/hda3 287 2741 19719787+ 83 LinuxDisk /dev/sda: 42.9 GB, 42949410816 bytes64 heads, 32 sectors/track, 40959 cylindersUnits = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System/dev/sda1 1 9538 9766896 83 Linux/dev/sda2 9539 19076 9766912 83 Linux/dev/sda3 19077 28614 9766912 83 Linux/dev/sda4 28615 40059 11719680 5 Extended/dev/sda5 28615 29592 1001456 83 Linux/dev/sda6 29593 30570 1001456 83 Linux/dev/sda7 30571 31548 1001456 83 Linux/dev/sda8 31549 35364 3907568 83 Linux/dev/sda9 35365 39180 3907568 83 Linux5、映射裸设备映射裸设备，分别在两个节点上配置，在这里映射的是扩展分区，而不是主分区，主分区是用来做ASM磁盘组安装数据库的。rootrac1 # vi /etc/sysconfig/rawdevices/dev/raw/raw1 /dev/sda5/dev/raw/raw2 /dev/sda6/dev/raw/raw3 /dev/sda7修改权限rootrac1 # vi /etc/udev/permissions.d/50-udev.permissionsraw/*:oracle:oinstall:0660启动并验证，两台机器rootrac1 # /etc/init.d/rawdevices startAssigning devices: /dev/raw/raw1 - /dev/sda5/dev/raw/raw1: bound to major 8, minor 5 /dev/raw/raw2 - /dev/sda6/dev/raw/raw2: bound to major 8, minor 6 /dev/raw/raw3 - /dev/sda7/dev/raw/raw3: bound to major 8, minor 7done 在rac2节点上启动并验证rootrac2 # /etc/init.d/rawdevices startAssigning devices: /dev/raw/raw1 - /dev/sda5/dev/raw/raw1: bound to major 8, minor 5 /dev/raw/raw2 - /dev/sda6/dev/raw/raw2: bound to major 8, minor 6 /dev/raw/raw3 - /dev/sda7/dev/raw/raw3: bound to major 8, minor 7done6、安装ASM按照顺序安装以下三个包，两个节点都要安装。首先oracleasm是要和系统内核版本一致的，查看linux内核版本。rootrac1 # uname -aLinux rac1 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux软件包oracleasm的版本必须和内核版本一致：oracleasm-support-2.1.7-1.el5.x86_64.rpmoracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64.rpmoracleasmlib-2.0.4-1.el5.x86_64.rpm按顺序安装三个软件包rootrac2 # rpm -ivh oracleasm-support-2.1.7-1.el5.x86_64.rpm warning: oracleasm-support-2.1.7-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159Preparing. # 100% 1:oracleasm-support # 100%rootrac2 # rpm -ivh oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64.rpm warning: oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159Preparing. # 100% 1:oracleasm-2.6.18-194.el# 100%rootrac2 # rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm warning: oracleasmlib-2.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159Preparing. # 100% 1:oracleasmlib # 100%创建asm磁盘，首先创建oracle用户和 oinstall组，两个节点都做，并且要保证两台节点的用户和组的id是一致的rootrac1 # groupadd oinstallrootrac1 # groupadd dbarootrac1 # useradd -g oinstall -G dba oraclerootrac1 # passwd oracleChanging password for user oracle.New UNIX password: BAD PASSWORD: it is too simplistic/systematicRetype new UNIX password: passwd: all authentication tokens updated successfully.查看用户和组id是否一致，或者在创建用户和组的时候直接指定用户和组的id。例如：Useradd -u（指定uid）。rootrac1 cluster# id oracleuid=500(oracle) gid=500(oinstall) groups=500(oinstall),501(dba)rootrac2 cluster# id oracleuid=500(oracle) gid=500(oinstall) groups=500(oinstall),501(dba)创建ASM，需要两个节点都要做rootrac1 # /etc/init.d/oracleasm configureConfiguring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets (). Hitting without typing ananswer will keep that current value. Ctrl-C will abort.Default user to own the driver interface oracle: oracleDefault group to own the driver interface oinstall: oinstallStart Oracle ASM library driver on boot (y/n) y: yScan for Oracle ASM disks on boot (y/n) y: yWriting Oracle ASM library driver configuration: doneInitializing the Oracle ASMLib driver: OK Scanning the system for Oracle ASMLib disks: OK 接下来告诉 ASMLib 驱动程序要使用哪些磁盘。注意，这些磁盘是不包含任何内容。命令格式：/etc/init.d/oracleasm createdisk DISK_NAME device_name这里只在其中一个节点中配置即可，另外一个基点能够自动检测到配置rootrac1 # /etc/init.d/oracleasm createdisk VOL1 /dev/sda1Marking disk VOL1 as an ASM disk: OK rootrac1 # /etc/init.d/oracleasm createdisk VOL2 /dev/sda2Marking disk VOL2 as an ASM disk: OK rootrac1 # /etc/init.d/oracleasm createdisk VOL3 /dev/sda3Marking disk VOL3 as an ASM disk: OK rootrac1 # /etc/init.d/oracleasm enableWriting Oracle ASM library driver configuration: doneInitializing the Oracle ASMLib driver: OK Scanning the system for Oracle ASMLib disks: OK rootrac2 # /etc/init.d/oracleasm enableWriting Oracle ASM library driver configuration: doneInitializing the Oracle ASMLib driver: OK Scanning the system for Oracle ASMLib disks: OK 在两个节点上扫描ASM磁盘rootrac1 # /etc/init.d/oracleasm listdisksVOL1VOL2VOL3rootrac2 # /etc/init.d/oracleasm listdisksVOL1VOL2VOL3二 、安装oracle cluster1、安装依赖包安装oracle集群所需要的软件包，两个节点都要安装。Rpm qa |grep compatcompat-boost-1331-1.33.1-5.0.1.el4compat-libgcc-296-2.96-132.7.2compat-openldap-2.1.30-12.el4compat-dapl-2.0.15-1.el4compat-gcc-32-c+-3.2.3-47.3compat-libstdc+-33-3.2.3-47.3compat-db-4.1.25-9compat-libcom_err-1.0-5compat-libstdc+-296-2.96-132.7.2compat-gcc-32-3.2.3-47.3openmotifbinutils-.6-5.el5control-center-2.16.0-14.el5gcc-4.1.2-14.el5gcc-c+-4.1.2-14.el5glibc-2.5-18libstdc+libstdc+-develmake-3.81-1.12、修改内核参数修改linux内核参数，两个节点都要修改rootrac1 # cat /etc/sysctl.conf /etc/security/limits.conf /etc/pam.d/login /etc/rc.d/rc.local EOFmodprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180EOF6、添加环境变量Oracle软件环境配置创建安装目录,两个节点都要创建。mkdir -p /oracle/app/oracle/10g/crs_1mkdir -p /oracle/app/oracle/10g/db_1chown -R oracle:oinstall /oracle/app/oracle/10g/crs_1 chown -R oracle:oinstall /oracle/app/oracle/10g/db_1chmod -R 775 /oracle/app/oracle/10g/crs_1 chmod -R 775 /oracle/app/oracle/10g/db_1oracle 用户的环境变量在oracle用户下，编辑/home/oracle/.bash_profile添加如下内容：在rac1机器上修改export ORACLE_SID=oradb1export ORACLE_BASE= /oracle/app/oracle/10gexport CRS_HOME=$ORACLE_BASE/product/db_1export ORACLE_HOME= /oracle/app/oracle/10g/crs_1export LD_L