solaris VCS启动故障处理.doc

一、背景：一台M8000服务器的一个domain wycaiji与另一台M8000的一个domain wangyou组成一个VCS集群，2012年底的一次割接中出现domain wycaiji无法启动VCS故障，wangyou单节点启动正常。二、信息收集及分析： 1、VCS手动命令启动过程报错如下： wycaiji gab: GAB ERROR V-15-1-20109 Port h registration failed, GAB not configured2、确定主机硬件工作状态，保证无硬件故障：3、通过对照wangyou和wycaiji的网卡启动情况：发现VCS正常的那台主机wangyou的心跳网卡bge2、bge6状态为已启动状态，而VCS异常的wycaiji主机的心跳网卡bge2、bge6均未启动。4、 通过对照wangyou和wycaiji的VCS：发现VCS正常的那台主机wangyou的gabconfig状态正常，而VCS异常的wycaiji主机的gabconfig输出为空。wycaiji:/#export PATH=$PATH:/usr/sbin:/sbin:/opt/VRTS/bin:/opt/VRTSvcs/binwycaiji:/#gabconfig -aGAB Port Memberships=wangyou:/#gabconfig -aGAB Port Memberships=Port a gen 50c70a membership 0Port h gen 50c70e membership 0 通过以上输出也能看到心跳网络不正常。初步推断VCS启动异常与心跳网卡未自动启动有关。 三、故障处理过程 1、 检查VCS有问题的主机wycaiji以下文件是否存在：wycaiji:/#more /etcttabset-node wycaijiset-cluster 34549link bge2 /dev/bge:2 - ether - -link bge6 /dev/bge:6 - ether - -link-lowpri bge4 /dev/bge:4 - ether - -set-timer peertrouble:400wycaiji:/#more /etcthosts0 wangyou1 wycaijiwycaiji:/etc#more gabtab/sbin/gabconfig -c -n22、 手动启动网卡，使之处于工作状态：wycaiji:/#ifconfig bge2 plumbwycaiji:/#ifconfig bge6 plumbwycaiji:/#ifconfig -alo0: flags=2001000849 mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 bge0: flags=209040843 mtu 1500 index 2 inet 10.223.3.50 netmask ffffffc0 broadcast 10.223.3.63 groupname ipmp1 ether 0:b:5d:e0:58:36 bge0:1: flags=201000843 mtu 1500 index 2 inet 10.223.3.49 netmask ffffffc0 broadcast 10.223.3.63bge2: flags=201000842 mtu 1500 index 5 inet 0.0.0.0 netmask 0 ether 0:b:5d:e0:57:6c bge4: flags=209040843 mtu 1500 index 3 inet 10.223.3.51 netmask ffffffc0 broadcast 10.223.3.63 groupname ipmp1 ether 0:b:5d:e0:52:cc bge6: flags=201000842 mtu 1500 index 6 inet 0.0.0.0 netmask 0 ether 0:b:5d:e0:53:22 sppp0: flags=10010008d1 mtu 1500 index 4 inet 192.168.10.2 - 192.168.10.1 netmask ffffff00 ether 0 3、 手动启动VCS，过程如下：wycaiji:/#lltconfig -cksh: lltconfig: not foundwycaiji:/#export PATH=$PATH:/usr/sbin:/sbin:/opt/VRTS/bin:/opt/VRTSvcs/binwycaiji:/#lltconfig -cNov 20 19:39:02 wycaiji llt: LLT INFO V-14-1-10483 16-bit cluster ID (34549) set.Updating protocol version from 3.7 to 4.0wycaiji:/#gabconfig -c -n2Nov 20 19:39:55 wycaiji llt: LLT INFO V-14-1-10499 recvarpack link 2 for node 0 addr change from 00:00:00:00:00:00 to 00:0B:5D:E0:55:78Nov 20 19:39:55 wycaiji llt: LLT INFO V-14-1-10499 recvarpack link 0 for node 0 addr change from 00:00:00:00:00:00 to 00:0B:5D:E0:54:DANov 20 19:39:55 wycaiji gab: GAB INFO V-15-1-20026 Port a registration waiting for seed port membershipNov 20 19:39:55 wycaiji llt: LLT INFO V-14-1-10499 recvarpack link 1 for node 0 addr change from 00:00:00:00:00:00 to 00:0B:5D:E0:53:1ENov 20 19:39:57 wycaiji llt: LLT INFO V-14-1-10024 link 1 (bge6) node 0 activeNov 20 19:39:57 wycaiji llt: LLT INFO V-14-1-10024 link 0 (bge2) node 0 activeNov 20 19:39:58 wycaiji llt: LLT INFO V-14-1-10024 link 2 (bge4) node 0 activeNov 20 19:40:02 wycaiji gab: GAB INFO V-15-1-20036 Port a gen 50c70b membership 01 wycaiji:/#hastartwycaiji:/#4、 检查VCS状态：wycaiji:/# hastatus -sum- SYSTEM STATE- System State Frozen A wangyou RUNNING 0 A wycaiji RUNNING 0 - GROUP STATE- Group System Probed AutoDisabled State B sg_NIC wangyou Y N ONLINE B sg_NIC wycaiji Y N ONLINE B sg_informix wangyou Y N ONLINE B sg_informix wycaiji N N OFFLINE B sg_wyapp wangyou Y N ONLINE B sg_wyapp wycaiji Y N OFFLINE - RESOURCES NOT PROBED- Group Type Resource System D sg_informix Informix informix_nios wycaiji wycaiji:/#hastatusattempting to connect.attempting to connect.connectedgroup resource system message - - - - wangyou RUNNING wycaiji RUNNING sg_NIC wangyou ONLINE sg_NIC wycaiji ONLINE -sg_informix wangyou ONLINE sg_wyapp wycaiji OFFLINE sg_wyapp wangyou ONLINE nic_bge04 wangyou ONLINE nic_bge04 wycaiji ONLINE - phantom_NIC wangyou ONLINE phantom_NIC wycaiji ONLINE dg_niosdb wangyou ONLINE dg_niosdb wycaiji OFFLINE ip_48 wangyou ONLINE - ip_48 wycaiji OFFLINE informix_nios wangyou ONLINE mnt_informix wangyou ONLINE mnt_informix wycaiji OFFLINE proxy_nic_db wangyou ONLINE - proxy_nic_db wycaiji ONLINE vol_informix wangyou ONLINE vol_informix wycaiji OFFLINE dg_nioscj wycaiji OFFLINE dg_nioscj wangyou ONLINE - ip_41 wycaiji OFFLINE ip_41 wangyou ONLINE mnt_nioscj wycaiji OFFLINE mnt_nioscj wangyou ONLINE proxy_ni