硬件更换流程.doc

8DA 22501 4008 ASCVQ A1430 HLR 硬件更换流程文档编号8DA 22501 4008 ASCVQ文档版本V2.5适应的运营商CMCC/UNICOM安全性风险高作 者张园庆联系方式产 品A1430 HLR适应版本R2, R3文档版本历史版本日期描述作者1.02006-07-29Creation of the document张园庆1.12006-07-29Update赵荣东1.22007-08-28Modify P4 红色字体张园庆1.32008-07-28Update key procedure.Lin Chunyi1.42009-05-10Update detailsLIN Chunyi2.02009/5/26updateHua youping2.12009/7/13增加微动开关的文字说明Wang peng2.22009/10/30增加lock板卡后需检查的状态Wang peng2.32009/11/23增加F300更换流程Wang peng2.42009/11/27updateLin chunyi2.52009/12/1Add checkpoint before pilot replacementZhao rongdong2.62010-2-9Update procedure for checking hardware clock and stopping DB clusterZhu meng1. Control station 替换准备工作： 1) 用df命令检查各目录硬盘空间占用率小于85%2) 更换硬件前检查系统中是否有自动备份在运行，如果有，请停止自动备份的运行。3) system , platform, all_data, mysql 数据备份。4) emergency restore 窗口已经下载。5) 新的pilot 备板BIOS参数已经重定向设置成通过COM接口连接到IOLAN输出，并且不从硬盘启动。检查新板卡微动开关的设置和下图一致：注意：只有SW4的第二个微动需要设置成on，其他所有的微动都是off。注意：以下图片的微动开关方向和板卡上的微动开关方向有可能不一致，以板卡上的微动开关的说明为准，例如：微动开关在板卡上的分布如下：6) PCFS通过IOLAN用telnet连接到需要更换的PILOT板7) 注意静电防护，比如带上静电手套等等更换过程1.1 lock和set equipment off standby pilot(1)在active pilot上执行fw lsta命令检查所有板卡运行状态，确认所有板卡的状态都是好的：roothlrsz5stb:# fw lsta* FWLSTA *host_name oper_state syst_state susp_req version STATION_A enabled standby no r3.310.1303-01 STATION_B enabled active no r3.310.1303-01 STATION_C enabled active no r3.310.1303-01 STATION_D enabled active no r3.310.1303-01 STATION_E enabled active no r3.310.1303-01 STATION_F enabled active no r3.310.1303-01 STATION_G enabled active no r3.310.1303-01 STATION_H enabled active no r3.310.1303-01 STATION_I enabled active no r3.310.1303-01 STATION_J enabled active no r3.310.1303-01 STATION_K enabled active no r3.310.1303-01 STATION_L enabled active no r3.310.1303-01 STATION_M disabled not_installed noSTATION_N disabled not_installed noSTATION_O disabled not_installed noSTATION_P disabled not_installed noSTATION_Q disabled not_installed noSTATION_R disabled not_installed noSTATION_S disabled not_installed noSTATION_T disabled not_installed noSTATION_U disabled not_installed noSTATION_V disabled not_installed noSTATION_W disabled not_installed noSTATION_X disabled not_installed noroothlrsz5stb:#在pilot上执行fw restart x (x为standby pilot)重启standby pilot执行fw lsta命令检查standby pilot启动完成，确认active / standby pilot。在active pilot上进入buii，执行命令：get staPILOT/*(*);检查standby pilot的如下状态参数为： operational_state = enabled, system_state = standby, 检查active pilot的如下状态参数为： operational_state = enabled, system_state = active, get logical_nodeNGHLR/A(*);get logical_nodeNGHLR/B(*);检查standby pilot的如下状态参数为： functional_name = Pilot, adm_state = unlocked, op_state = enabled, failed = no, in_test = no, power_off = no, offline = no, dependency = no, degraded = no, not_installed = no, st_status = hot_standby, node_type = tomas_station, node_subtype = tomas_pilot_station,检查active pilot的如下状态参数为： adm_state = unlocked, op_state = enabled, failed = no, in_test = no, power_off = no, offline = no, dependency = no, degraded = no, not_installed = no, st_status = providing_service, node_type = tomas_station, node_subtype = tomas_pilot_station,(2)在active pilot上使用fw lock X(X=A or B)命令锁定standby pilot：roothlrsz5stb:# fw lock A (假设A为standby pilot的情况)- OK, connection active - Number of clients = 14/* Answer to request #1 = */ACTION_RSP logical_node NGHLR/A(lock();roothlrsz5stb:#(3)standby pilot板卡被锁定后检查以下状态是否正确：roothlrsz5stb:# fw lsta* FWLSTA *host_name oper_state syst_state susp_req version STATION_A disabled not_installed no r3.310.1303-01 STATION_B enabled active no r3.310.1303-01 STATION_C enabled active no r3.310.1303-01 STATION_D enabled active no r3.310.1303-01 STATION_E enabled active no r3.310.1303-01 STATION_F enabled active no r3.310.1303-01 STATION_G enabled active no r3.310.1303-01 STATION_H enabled active no r3.310.1303-01 STATION_I enabled active no r3.310.1303-01 STATION_J enabled active no r3.310.1303-01 STATION_K enabled active no r3.310.1303-01 STATION_L enabled active no r3.310.1303-01 STATION_M disabled not_installed noSTATION_N disabled not_installed noSTATION_O disabled not_installed noSTATION_P disabled not_installed noSTATION_Q disabled not_installed noSTATION_R disabled not_installed noSTATION_S disabled not_installed noSTATION_T disabled not_installed noSTATION_U disabled not_installed noSTATION_V disabled not_installed noSTATION_W disabled not_installed noSTATION_X disabled not_installed noroothlrsz5stb:#roothlrsz5stb:# buii- OK, connection active - Number of clients = 14BUI get staPILOT/A(*); (假设A为standby pilot的情况)Answer to request #1 =GET_RSP sta PILOT/A ( /* Attributes : */ creation_date = 2009/08/28 10:54:32, adm_upd_date = 2009/08/28 10:54:32, tsap = 65, tsap_mirror = 66, hostname = STATION_A, profile_name = _pilot_, type_sta = pilot, syst_upd_date = 2009/10/30 16:05:12, operational_state = disabled, system_state = not_installed, suspend_requested = no, version = r3.310.1303-01 ) ;The request number 1 has been sent to localhostBUI get logical_nodeNGHLR/A(*); (假设A为standby pilot的情况)Answer to request #2 =GET_RSP logical_node NGHLR/A ( /* Attributes : */ creation_date = 2006/06/09 21:52:59, adm_upd_date = 2007/08/10 02:27:12, syst_upd_date = 2009/10/30 16:05:13, functional_name = Pilot, adm_state = locked, op_state = disabled, failed = no, in_test = no, power_off = no, offline = yes, dependency = no, degraded = no, not_installed = no, st_status = cold_standby, node_type = tomas_station, node_subtype = tomas_pilot_station, node_name = STATION_A, tsap = 65, memory_size = 4, cpu_number = 2,(4)给被锁定的pilot下电：BUI action logical_nodeNGHLR/A(set_equipment_off(); (假设A为standby pilot的情况)Answer to request #3 =ACTION_RSP logical_node NGHLR/A(set_equipment_off();The request number 3 has been sent to localhostBUI(5)下电完成后检查以下红色标示部分状态是否正常：roothlrsz5stb:# fw lsta* FWLSTA *host_name oper_state syst_state susp_req version STATION_A disabled not_installed no r3.310.1303-01 STATION_B enabled active no r3.310.1303-01 STATION_C enabled active no r3.310.1303-01 STATION_D enabled active no r3.310.1303-01 STATION_E enabled active no r3.310.1303-01 STATION_F enabled active no r3.310.1303-01 STATION_G enabled active no r3.310.1303-01 STATION_H enabled active no r3.310.1303-01 STATION_I enabled active no r3.310.1303-01 STATION_J enabled active no r3.310.1303-01 STATION_K enabled active no r3.310.1303-01 STATION_L enabled active no r3.310.1303-01 STATION_M disabled not_installed noSTATION_N disabled not_installed noSTATION_O disabled not_installed noSTATION_P disabled not_installed noSTATION_Q disabled not_installed noSTATION_R disabled not_installed noSTATION_S disabled not_installed noSTATION_T disabled not_installed noSTATION_U disabled not_installed noSTATION_V disabled not_installed noSTATION_W disabled not_installed noSTATION_X disabled not_installed noroothlrsz5stb:# buii- OK, connection active - Number of clients = 14BUI get staPILOT/A(*); (假设A为standby pilot的情况)Answer to request #1 =GET_RSP sta PILOT/A ( /* Attributes : */ creation_date = 2009/08/28 10:54:32, adm_upd_date = 2009/08/28 10:54:32, tsap = 65, tsap_mirror = 66, hostname = STATION_A, profile_name = _pilot_, type_sta = pilot, syst_upd_date = 2009/10/30 16:05:12, operational_state = disabled, system_state = not_installed, suspend_requested = no, version = r3.310.1303-01 ) ;The request number 1 has been sent to localhostBUI get logical_nodeNGHLR/A(*); (假设A为standby pilot的情况)Answer to request #2 =GET_RSP logical_node NGHLR/A ( /* Attributes : */ creation_date = 2006/06/09 21:52:59, adm_upd_date = 2007/08/10 02:27:12, syst_upd_date = 2009/10/30 16:13:33, functional_name = Pilot, adm_state = locked, op_state = disabled, failed = no, in_test = no, power_off = no, offline = yes, dependency = no, degraded = no, not_installed = yes, st_status = cold_standby, node_type = tomas_station, node_subtype = tomas_pilot_station, node_name = STATION_A, tsap = 65, memory_size = 4, cpu_number = 2,注意：standby pilot被lock住后状态必须是not_installed，如果standby pilot被lock住后状态仍然是standby，请停止当前的所有操作，并立即联系相关区域的TAC负责人！1.2 替换硬件 拨出要替换的板卡：注意先松开板卡下面的扳手，等蓝灯常亮之后再拔出； 插入新的PILOT后，立即在PCFS上通过IOLAN用telnet连接到需要更换的PILOT板，按F2进入BIOS。如果进入BIOS没有成功，应立即启动该PILOT的reset按钮，并在PCFS上按F2进入BIOS，记录该PILOT的两个MAC地址，设置该板卡的BIOS，BIOS参数设置见附录4.1，不要退出。 在新插入的PILOT板BIOS中检查并修改确认系统不从硬盘启动，在Hard Drive前加上“！”，并保存退出。注意：如果有CF卡，需要将CF卡启动放在硬盘启动的前面并去掉卡启动前面的”!”，如下图：Hard Drive下面TRANSCEND-(SM)表示CF卡启动1.3 修改备用侧的MAC 地址登录到active pilot： 在Linux 下， 在#后输入以下命令并回车/usr/nectar/bin/nse/nse_clientupdate station-name 说明： station-name is standby pilot; example: station_A. or .station_B 选择2 修改ethernet address 并回车。 择对应需要修改的pilot 名称并回车。 按照提示替换两个MAC 地址。 确认MAC地址是否正确。如果检查一致，输入字母Y并回车。 输入q结束配置。 1.4 改备用侧PILOT的BIOS执行restore硬件reset刚插入的板卡 按住PCFS的F2，进入BIOS 设置。 在BOOT 选项中将ETH1 和ETH2 移到 Hard drive 之前。 保存退出，备用侧PILOT然后开始自动RESTORE。 等到restore结束（大约30分钟），会出现以下信息：Do you want to reboot 输入Y，回车。（请马上按住F2，进入BIOS设置）1.5 修改备用侧PILOT的BIOS 按住终端电脑的F2，进入BIOS 设置。 检查BIOS里的时间设置，如果和active pilot 时间不一致，请修改过来 在BOOT 选项中将ETH1 和ETH2 移到 Hard drive 之后。在Hard Drive前去掉“！”， 注意：如果pilot板卡安装了CF卡，将CF卡前面的”!”去掉，并放在硬盘启动的前面！，设置完毕之后，如下图：TRANSCEND-(SM)放在第一启动位卡启动FUJITSU MHV2060AS-(PM放在第二启动位)硬盘启动网卡启动放在后面(IBA GE Slot 012.)注意：在极少数情况下，Hard Drive下的设备的第二和第三位置显示可能有问题，表现为该设备是禁止引导的（前面有“！”号），这时使用“shift+1”无法去除前面的“！”号，但是在第一个位置的设备能正常添加和删除“！”号，如果遇到这种情况，请把3个设备都轮流移动（使用“+”和“-”号操作）到第一个位置，把需要的状态修改好后（加“！”或删除“！”）再移动到第二和第三位置。 保存退出， 开始从硬盘起。 如果更换的是用一块新的或格式化过的PILOT，需要执行下面命令创建 partition 15。在备用侧PILOT进入Linux 后， 在#后执行mke2fs -j /dev/discs/disc0/part15 ， 然后执行reboot 重启。启动完毕，创建一个目录：= mkdir -p /var/local/mds/replication 重新进入Linux 后，在#后执行以下命令：=/usr/nectar/conf/bin/necConfig -fm=/usr/nectar/bin/nse/nse_insertdisk -y1.6 Set equip on和unlock 备用板卡在active Pilot station上执行命令：如果Station_A为Standby Pilot Station:=buii=action logical_nodeNGHLR/A(set_equipment_on();=fw unlock A如果Station_B为Standby Pilot Station:=buii=action logical_nodeNGHLR/B(set_equipment_on();=fw unlock B如果因备用侧存在告警而状态变成Not installed, 需要在主用侧PILOT上执行命令fw clear X (X为备用侧PILOT字母，A或B)。将备用侧PILOT的告警消除后才能启动成功。1.7 等备侧变成standby后检查系统状态=fw lsta 各station状态都为active =fw lsndb 各个DB cluster 都为started. =stg 结果显示应该为Yes. 若为NO，则需执行命令：st yes1.8 完成以下检查在buii下执行以下两条命令:get logical_nodeNGHLR/A(*);get logical_nodeNGHLR/B(*);检查命令执行后的adm_state状态必须为unlock，如果状态为lock，必须使用fw unlock X(X=A or B)解锁该板卡。1.9 使用date命令检查active pilot和standby pilot时间是否一致1.10 做System backup1.11 如果更换前停止了自动备份的运行，请恢复自动备份的自动运行。2. DB station 替换准备工作： 1) 做system backup。2) 更换硬件前检查系统中是否有自动备份在运行，如果有，请停止自动备份的运行。3) 如果需要更换的DB station运行正常，进行 all_data, mysql 数据备份。否则，不进行alldata和mysql数据备份。4) 检查系统中system，all_data, mysql backup都已经存在5) emergency restore 窗口已经下载。6) 新的DB备板已经格式化，并且BIOS参数已经重定向设置成通过ShMC输出，并且不从硬盘启动。7) 注意静电防护，比如带上静电手套等等更换过程：2.1 确认系统状态1） 用fw lsta & fw lsndb 检查系统状态， 除需要替换的station其余状态正常。2） 确定需要替换扳子的物理位置。2.2 创建格式化文件登录到active station中：Example:（以STATION_C为例，下列命令中STATION_C必须大写）rootlalxhlr04b:# touch /spdata/dbsupervisor/formatSTATION_CNeededstation_c即为需要被替换的DB板卡2.3 DB板卡下电1) 在Active Poilt Station上使用fw stop 命令，将DB板卡上的cluster停掉，可以在/usr/dhafw/data/mysql/config.ini里找到对应值，以station_c为例 rootshxhlr23b:/# fw stop 32)在Active Pilot Station上lock DB Station_C:rootshxhlr23b:/# buiiBUIaction logical_nodeNGHLR/C(lock();3) 在Active Pilot Station上Power off此DB Station_C:rootshxhlr23b:/# buiiBUI action logical_nodeNGHLR/C(set_equipment_off();2.4 替换硬件用新的DB STATION更换，更换前检查新的DB板微动开关的设置与被替换下来的的DB版微动开关要一致：微动开关位置见下图，注意与原板卡设置成一样，每个SW有编号14个开关,设置为ON或者OFF。2.5 DB板卡上电1) 在Active Pilot Station上set on此DB Station:rootshxhlr23b:/# buiiBUI action logical_nodeNGHLR/C(set_equipment_on();2) unlock 此DB stationrootshxhlr23b:/# buiiBUIaction logical_nodeNGHLR/C(unlock();2.6 检查BIOS设置通过SHMC登录该Station的BIOS:按照附录4.2设置该Station的BIOS.设置完毕，保存退出。2.7 检查DB板状态1) 检查此DB station状态rootshxhlr23b:/# fw lsta* FWLSTA *hostname oper_state syst_state susp_req version STATION_A enabled standby no r3.006.0003-02.00 STATION_B enabled active no r3.006.0003-02.00 STATION_C enabled active no r3.006.0003-02.00 STATION_D enabled active no r3.006.0003-02.00 STATION_E enabled active no r3.006.0003-02.00 STATION_F enabled active no r3.006.0003-02.00 STATION_G enabled active no r3.006.0003-02.00 STATION_H enabled active no r3.006.0003-02.00 STATION_I enabled active no r3.006.0003-02.00 STATION_J enabled active no r3.006.0003-02.00 STATION_K enabled active no r3.006.0003-02.00 STATION_L enabled active no r3.006.0003-02.00 STATION_M disabled not_installed no STATION_N disabled not_installed no STATION_O disabled not_installed no STATION_P disabled not_installed no STATION_Q disa