Sun cluster 3x 培训(规划、安装、配置和管理)

1、Sun cluster 3.x 规划、安装、配置和管理双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划2 2、操作系统的规划、操作系统的规划 3 3、磁盘管理工具的规划、磁盘管理工具的规划双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划Sun cluster 3.1 系统硬件结构 主机、备机的硬件配置要求相同，如果不同，二者的处理能力不要相差甚远；主机、备机的硬件配置要求相同，如果不同，二者的处理能力不要相差甚远； 主机、备机的

2、体系结构必须相同主机、备机的体系结构必须相同; ; 最多可以支持到最多可以支持到1616个主机节点；个主机节点； 支持支持 solaris 8solaris 8、solaris 9; solaris 9; Sun cluster 3.1 系统硬件结构Sun clusterSun cluster平台最小硬件组成：平台最小硬件组成：1 1、管理工作站、管理工作站（administration workstation）2 2、串口集线器、串口集线器（Terminal Concentrator）3 3、至少两台服务器、至少两台服务器（Two Host (Up to 16)）4 4、每台服务器至少配置一

3、个公网网口、每台服务器至少配置一个公网网口（One or more public network interfaces per host）5 5、至少一条心跳线、至少一条心跳线(A private cluster transport interface)6、可以提供双端口的磁盘阵列可以提供双端口的磁盘阵列(Dual hosted, mirrored disk storage) (以T3单端口磁盘阵列为例说明)主机节点冗余Redundant Servers每个节点本身的一些部件是冗余的；每个节点本身的一些部件是冗余的；节点之间互为备份是冗余的；节点之间互为备份是冗余的；每个节点对外提供服务的网口

4、是冗余的Redundant Public Networks两个节点相互通信的网口是冗余的Redundant Private Networks Only one of the links is used at a time. If the primary link fails, the cluster software automatically switches to the backup link.This is transparent to all cluster applications.具体项目举例：具体项目举例：磁盘系统是冗余的Redundant DataSun cluster3.x

5、 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：投票机制：投票机制：思考思考1：如果节点之间失去通讯怎么办？如果节点之间失去通讯怎么办？Sun cluster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：投票机制：投票机制：1、每个节点都会认为对方节点是正常的，这样，可能导致数据丢失；、每个节点都会认为对方节点是正常的，这样，可能导致数据丢失；2、This is called split-brain operation. Sun cluster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念

6、介绍：投票机制：投票机制：1 1、两个节点首先尝试保留指定的投票设备；、两个节点首先尝试保留指定的投票设备；2 2、第一个、第一个获得投票设备的节点将作为获得投票设备的节点将作为cluster成员保留在成员保留在cluster中；中；3、在竞争中，失去投票设备的节点将退出、在竞争中，失去投票设备的节点将退出cluster；Sun cluster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：投票机制：投票机制：Sun cluster架构引入了一种投票机制架构引入了一种投票机制“voting system”，在这个投票系统中：在这个投票系统中：1、每个节点拥

7、有一票（one voteone vote）；2、被指定为投票设备（quorum devices）的共享硬盘拥有一票；3、整个系统必须拥有多数票（majority）时才能形成cluster，正常运行； (more than 50 percent of all possible votes present)Sun cluster3.x 软件架构及切换原理软件架构及切换原理 选取投票设备的一般原则选取投票设备的一般原则（general rules） ：1、在两个节点的、在两个节点的cluster环境中，环境中，quorum设备必须能够被多个节点访问设备必须能够被多个节点访问；2、quorum设备的信

8、息保存在设备的信息保存在CCR数据库中；数据库中；3、被指定为、被指定为quorum设备的磁盘仍然能够正常使用为数据盘；设备的磁盘仍然能够正常使用为数据盘；4、cluster环境中环境中quorum设备的最优设备的最优/最多数量为节点数减最多数量为节点数减1：N-1；5、在多于两个节点的、在多于两个节点的cluster环境中，环境中，quorum设备的选用不是必需的，但建议选用；设备的选用不是必需的，但建议选用；6、quorum设备在设备在cluster安装配置过程中指定；安装配置过程中指定；7、指定、指定quorum设备时，使用设备时，使用did命名系统下的设备；命名系统下的设备；Sun c

9、luster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：Persistent Reservations and Reservation Keys 每一个节点在投票设备上分配了一个每一个节点在投票设备上分配了一个64-bit的的key;Sun cluster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：Persistent Reservations and Reservation Keys如果因为某种原因，节点如果因为某种原因，节点2 2退出退出cluster系统，实系统，实际上，节点际上，节点1 1会从投票设备上

10、删除节点会从投票设备上删除节点2 2的的64-64-bitbit的的keykey；Sun cluster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：Persistent Reservations and Reservation KeysSCSI-2 Persistent Reservation Emulation The scenario presented in the previous section was actually using a method called SCSI-2 Persistent Reservation Emulation

11、 (PRE) to implement the reservation keys. PREs have the following characteristics. The persistent reservations are not supported directly by the SCSI command set. Reservation keys are written on “private cylinders” of the disk (cylinders that are not visible in the format command but are still direc

12、tly writable by Solaris OE). The reservation keys have no impact on using the disk as a regular data disk, where you will not see the private cylinders. The “race” (for example, in split brain scenario) is still decided by a normal SCSI-2 disk reservation. When you add a quorum device, the Sun Clust

13、er software automatically will use SCSI-2 PRE with that quorum device if there are exactly two sub-paths for the DID associated with that device.Sun cluster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：Persistent Reservations and Reservation KeysSCSI-3 Persistent Group ReservationWhen there are more th

14、an two nodes physically connected to the disk,SCSI-3 Persistent Group Reservations (PGR) are used. PGR have the following characteristics The persistent reservations are implemented directly by the SCSI-3 command set. Disk firmware itself must be fully SCSI-3 compliant. “Racing” to remove another no

15、des reservation key does not involve physical reservation of the disk.Sun cluster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：Persistent Reservations and Reservation KeysSCSI-3 Persistent Group Reservation four nodes are all physically connected to a quorum drive.Remember the single quorum drive will

16、have three quorum votes. Now imagine Nodes 1 and 3 can see each other over the transport and Nodes 2 and 4 can see each other over the transport. Sun cluster3.x 软件架构及切换原理软件架构及切换原理 重要模块及概念介绍：重要模块及概念介绍：Persistent Reservations and Reservation KeysSCSI-3 Persistent Group Reservation In each pair, the no

17、de with the lower reservation key will try to eliminate the reservation key of the other pair. The SCSI-3 protocol assures that only one pair will win. Because Nodes 2 and 4 have their reservation key eliminated, they cannot count the three votes of the quorum device. Because they fall below the nee

18、ded quorum, they will kernel panic. Cluster amnesia is avoided in the exact same way as in a two-node quorum device. If you now shut down the whole cluster, Node 2 and Node 4 could not count the quorum device because their reservation key is eliminated. They would have to wait for either Node 1 or N

19、ode 3 to join. One of those nodes could then add back reservation keys for Node 2 and Node 4.双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划软件、硬件平台配置：部件部件节点节点A A节点节点B B系统硬盘系统硬盘2*9G2*9G公网网卡公网网卡两块两块私网网卡私网网卡四口网卡一块四口网卡一块SCSISCSI卡卡两块两块共享磁盘共享磁盘2*Multipack，每个：4*18G（Raid0+1）操作系统操作系统Solaris 9磁盘管理软件磁盘管理软件Volume manager 3.2

20、双机软件双机软件SUN Cluster 3.1应用程序应用程序AGENT AGENT 软件软件NFSSUNW.nfs硬件互联要必须解决的一个关键问题：硬件互联要必须解决的一个关键问题：scsi-initiator-idscsi-initiator-id双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划硬件互联要必须解决的一个关键问题：硬件互联要必须解决的一个关键问题：scsi-initiator-idscsi-initiator-id双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划1 1、硬件互联的规划、硬件互联的规划修改修改scsi-initiator-id

21、的步骤：的步骤：Ok setenv auto-boot? false /default is trueOk reset-allOk probe-scsi-all /pci1f,2000/pci1/SUNW，isptwo4/pci1f,4000/pci1/SUNW，isptwo4双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划修改scsi-initiator-id的步骤：Ok nvedit 0: probe-all install-console banner1: cd /pci1f,2000/pci1/SUNW，isptwo4 /进入要操作的适配器；2: 6 scsi-ini

22、tiator-id integer-property /scsi前有空格；3: device-end /退出适配器；4: cd /pci1f,4000/pci1/SUNW，isptwo4 /进入要操作的适配器；5: 6 scsi-initiator-id integer-property /scsi前有空格；6: device-end 7: Ok nvstore 双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划修改修改scsi-initiator-idscsi-initiator-id的步骤：的步骤：Ok setenv use-nvramrc? true Ok setenv

23、use-nvramrc? true Ok reset-allOk reset-all如何确认：如何确认：方法一：ok cd /pci1f,4000/pci1/SUNW，isptwo4 /进入要操作的适配器；ok .properties 6 scsi-initiator-id方法二：系统再次启动时，会出现如下的信息：系统再次启动时，会出现如下的信息：isp0: initiator SCSI ID now 6说明修改成功！说明修改成功！ 双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划注意注意： 如果用如果用nvedit命令编辑上述过程有错误，用如下命令可以清命令编辑上述过程有错

24、误，用如下命令可以清空：空：Ok set-default nvramrc 在正式运行系统上在正式运行系统上任何时候都不要进行任何时候都不要进行set-defaults命令命令，这，这样将会清空关键的配置数据；样将会清空关键的配置数据； 双机系统的规划双机系统的规划关于主机关于主机mac地址的设置：地址的设置：ok setenv localmacaddress true 使得每个网卡都有不同的使得每个网卡都有不同的mac地址。地址。双机系统的规划双机系统的规划1 1、硬件互联的规、硬件互联的规划划记录好每个节点的记录好每个节点的macmac地址：地址：主机点主机点备节点备节点node01/eth

25、ernode02/ether8:0:20:d2:2b:7d8:0:20:d2:2c:41公网网线的做法：公网网线的做法： 从左到右的线序：从左到右的线序： 1 1 2 2 3 4 5 6 7 8 3 4 5 6 7 8橙白、橙、绿白、蓝、蓝白、绿、棕白、棕橙白、橙、绿白、蓝、蓝白、绿、棕白、棕橙白、橙、绿白、蓝、蓝白、绿、棕白、棕橙白、橙、绿白、蓝、蓝白、绿、棕白、棕双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划 私网网线的做法：私网网线的做法： 从左到右的线序：从左到右的线序： 1 1 2 2 3 4 5 6 7 8 3 4 5 6 7 8橙白、橙、绿白、蓝、蓝白、绿、棕

26、白、棕橙白、橙、绿白、蓝、蓝白、绿、棕白、棕绿白、绿、橙白、蓝、蓝白、橙、棕白、棕绿白、绿、橙白、蓝、蓝白、橙、棕白、棕双机系统的规划双机系统的规划1 1、硬件互联的规划、硬件互联的规划2 2、操作系统及相关软件的规划、操作系统及相关软件的规划双机系统的规划双机系统的规划2 2、操作系统及相关软件的规划、操作系统及相关软件的规划系统盘的规划；主机名/IP地址规划；cluster名标识定义；共享磁盘的规划；双机系统的规划双机系统的规划双机系统的规划双机系统的规划2 2、操作系统的规划、操作系统的规划系统盘的规划系统盘的规划1、根分区：、根分区： /dev/dsk/c0t0d0s0 / /剩余空间

27、剩余空间2、交换区：、交换区： /dev/dsk/c0t0d0s1 swap /物理内存的物理内存的23倍；倍；3、私有分区：、私有分区： /dev/dsk/c0t0d0s34、公有分区：、公有分区： /dev/dsk/c0t0d0s4 5、数据分区：、数据分区： /dev/dsk/c0t0d0s5 /data6、设备树：、设备树： /dev/dsk/c0t0d0s6 /globaldevices /至少至少512M7、预留分区：、预留分区： /dev/dsk/c0t0d0s7 /20MB；主机名主机名/IP/IP地址规划地址规划 1、主机名、主机名/IP规划：规划：双机系统的规划双机系统的规

28、划2 2、操作系统的规划、操作系统的规划主节点备节点/node0/node02资源组IP:00/nfsserver主机名/IP地址规划 3 3、私网（心跳）、私网（心跳）IPIP规划：规划：注：私网注：私网IPIP由由SUN CLUSTERSUN CLUSTER软件自己指定和维护，所有的平台都软件自己指定和维护，所有的平台都一样，最好不要改变！一样，最好不要改变！双机系统的规划双机系统的规划2 2、操作系统的规划、操作系统的规划cluster名标识定义:双机系统的规划双机系统的规划2 2、操作系统的规划、操作系统的规划主节点主节点 备节点备节点

29、 /node01 /node02 资源组资源组IP:00/nfsserverCluster标识：mycluster共享磁盘的规划双机系统的规划双机系统的规划2 2、操作系统的规划、操作系统的规划高可用服务高可用服务NFSNFS（网络文件系统）（网络文件系统）磁盘组nfsdg磁盘组包含磁盘磁盘组包含磁盘c1t1d0/c1t2d0RAID格式raid0+1 部署一套双机系统，一共需要五类软件：部署一套双机系统，一共需要五类软件： Solaris 9 U3 08/03 Sun Cluster 3.1 05/03 Sun Cluster 3.1 Agen

30、ts File system supported（可选） Unix file system (UFS) Veritas VxFS 3.5 Volume manager supported Solstice Volume Manager 4.2.1 Veritas Volume Manager 3.5 Veritas VxVM Cluster Functionality(for OPS/Oracle 9i RAC)双机系统的规划双机系统的规划2 2、sun cluster3.xsun cluster3.x相关软件的规划相关软件的规划3 3、磁盘管理工具的规划、磁盘管理工具的规划双机系统的规划双机

31、系统的规划3 3、磁盘管理工具的规划、磁盘管理工具的规划 veritas volume manager (svm) sun solstice disksuite (sds) cluster volume manager（cvm） 双机系统的规划双机系统的规划双机系统的安装双机系统的安装双机系统的安装双机系统的安装一、终端集线器的安装；一、终端集线器的安装；二、硬件互联；二、硬件互联；三、主机操作系统的安装；三、主机操作系统的安装；四、四、SUN CLUSTER 3.1软件的安装；软件的安装；五、磁盘管理工具安装；五、磁盘管理工具安装；六、六、AgentAgent软件的安装；软件的安装；七、应用

32、软件的安装七、应用软件的安装 ；双机系统的安装双机系统的安装Sun Cluster 3.1 software basic installation process 1. Repartition boot disks 2. Install the Solaris OE software 3. Configure the environment for the cluster host systems 4. Install Solaris OE patches 5. Install hardware-related patches 6. Install Sun Cluster 3.1 softwa

33、re on the first cluster node 7. Install Sun Cluster 3.1 software on the remaining nodes 8. Install any Sun Cluster software patches 9. Perform postinstallation checks and configuration双机系统的安装双机系统的安装 Sun Cluster Software Alternative Installation ProcessesThere are three different methods for installi

34、ng the Sun Cluster software.For instructional purposes, the exercises in this course use the manual method. You must have a good understanding of the basic installation process before attempting the more advanced methods.q Manual InstallationUse the Sun Cluster software scinstall utility to manually

35、 install the Sun Cluster software on each cluster host system. The Solaris OE software must be previously installed.q JumpStart Software InstallationIf you have an existing JumpStart software server, you can use the scinstall utility to include the Sun Cluster software installation in the existing J

36、umpStart configuration. The Sun Cluster CD-ROM image is copied to the JumpStart server. You can then automatically install the Solaris OE and the Sun Cluster software in a single operation.Note You must answer all of the scinstall questions just as you do during a manual installation. The resulting

37、information is stored and used during the JumpStart installation.q SunPlex Manager Installation You can use the SunPlex Manager application to install the Sun Cluster software. You must first install the Solaris OE and the SunPlex Manager software on each cluster node. The SunPlex Manager software p

38、ackages are automatically installed on each node during the manual Sun Cluster software installation process.双机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process3. Configure the environment for the cluster host systems node # vi /etc/nsswitch.conf.hosts: cluster files.node # vi /etc/hosts1

39、 localhost node1 loghost node5 admin00 tc00 nfsserver# vi /.rhosts+双机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process3. Configure the environment for the cluster host systemsConfiguring the User root EnvironmentThe root login environment s

40、hould include the following search path and man page information:PATH=$PATH:/usr/cluster/bin:/etc/vx/binMANPATH=$MANPATH:/usr/cluster/man:/usr/share/man:/opt/VRTS/man双机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process4. Installing Sun Cluster Software Node PatchesThe Sun Cluster software

41、nodes might require patches in the following areas:Solaris OE patchesStorage array interface firmware patchesStorage array disk drive firmware patchesVERITAS Volume Manager patchesSolstice DiskSuite software patches注意：You cannot install some patches, such as those for VERITAS Volume Manager and Sols

42、tice DiskSuite software, until after the volume management software installation is completed.双机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process4. Installing Sun Cluster Software Node PatchesPatch Installation WarningsBefore installing any patches, always do the following:Make sure all c

43、luster nodes have the same patch levels.Do not install any firmware-related patches without qualified assistance.Always obtain the most current patch information.Read all patch README notes carefully.双机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process4. Installing Sun Cluster Software Nod

44、e PatchesObtaining Patch InformationConsult http:/.Use the SunSolveSM PatchPro tool interactivelyRead current Sun Cluster software release notes双机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process6. Install Sun Cluster 3.1 software on the first cluster nodeq Complete the installation on th

45、e first node and run scinstall on all other nodes in parallelnode1# cd /suncluster_3_1/SunCluster_3.1/Sol_9/Toolsnode1# lsdefaults lib scinstallnode1# ./scinstall* Main Menu *Please select from one of the following (*) options:* 1) Establish a new cluster using this machine as the first node* 2) Add

46、 this machine as a node in an established cluster3) Configure a cluster to be JumpStarted from this install server4) Add support for new data services to this cluster node5) Print release information for this cluster node* ?) Help with menu options* q) QuitOption: 1双机系统的安装双机系统的安装Sun Cluster 3.1 host

47、 system installation process6. Install Sun Cluster 3.1 software on the first cluster nodeq Complete the installation on the first node and run scinstall on all other nodes in parallel* Establishing a New Cluster *This option is used to establish a new cluster using this machine as the first node in

48、that cluster.Once the cluster framework software is installed, you will be asked for the name of the cluster. Then, sccheck(1M) is run to test this machine for basic Sun Cluster pre-configuration requirements.After sccheck(1M) passes, you will be asked for the names of the other nodes which will ini

49、tially be joining that cluster. In addition, you will be asked to provide certain cluster transport configuration information.Press Ctrl-d at any time to return to the Main Menu.Do you want to continue (yes/no) yes? Software Package Installation Cluster Name Check Cluster Nodes Authenticating Reques

50、ts to Add Nodes Network Address for the Cluster Transport Point-to-Point Cables Cluster Transport Adapters and Cables Select the first cluster transport adapter to use: Select the second cluster transport adapter to use: Global Devices File System Each node in the cluster must have a local file syst

51、em mounted on /global/.devices/node before it can successfully participate as a cluster member. Since the nodeID is not assigned untilscinstall is run, scinstall will set this up for you. However, in order to do this, you must supply the name of either an already-mounted file system or raw disk part

52、ition at this time. This file system or partition should be at least 512 MB in size.If an already-mounted file system is used, the file system must beempty. If a raw disk partition is used, a new file system will be created for you.The default is to use /globaldevices.Is it okay to use this default

53、(yes/no) yes?双机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process6. Install Sun Cluster 3.1 software on the first cluster nodeq Complete the installation on the first node and run scinstall on all other nodes in parallel Automatic Reboot Confirmation Initial Cluster Setup This program has

54、detected that the cluster installmode attribute is still enabled. As such, certain initial cluster setup steps will be performed at this time. This includes adding any necessary quorumdevices, then resetting both the quorum vote counts and the installmode property.Please do not proceed if any additi

55、onal nodes have yet to join the cluster.Is it okay to continue (yes/no) yes?Do you want to add any quorum disks (yes/no) yes?Dual-ported SCSI-2 disks may be used as quorum devices in two-node clusters. However, clusters with more than two nodes require that SCSI-3 PGR disks be used for all disks wit

56、h more than twonode-to-disk paths. You can use a disk containing user data or one that is a member of a device group as a quorum device.Each quorum disk must be connected to at least two nodes. Please refer to the Sun Cluster documentation for more information on supported quorum device topologies.双

57、机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process9. Perform postinstallation checks and configurationPostinstallation Configuration After the initial Sun Cluster software installation, you must complete the following tasks Resetting Install ModeWhich global device do you want to use (d)?

58、 d3Is it okay to proceed with the update (yes/no) yes?scconf -a -q globaldev=d3Command completed successfully.Hit ENTER to continue:Do you want to add another quorum disk (yes/no)? noOnce the installmode property has been reset, this program will skip Initial Cluster Setup each time it is run again

59、in the future.However, quorum devices can always be added to the cluster using the regular menu options. Resetting this property fully activates quorum settings and is necessary for the normal and safe operation of thecluster.Is it okay to reset installmode (yes/no) yes?scconf -c -q resetCluster ini

60、tialization is complete.双机系统的安装双机系统的安装Sun Cluster 3.1 host system installation process9. Perform postinstallation checks and configurationPostinstallation Configuration After the initial Sun Cluster software installation, you must complete the following tasks q Time between all nodes in a cluster mu