R6217p1-Upgrade Guideline.doc

R6217p1 Upgrade Guideline V1.0VersionComments1.0First Draft一、 升级准备1. SEOS下载XCRP3Bootromof1272.binMinikernelnetbsd.min.v23.bz2.binSEOSSEOS-p1.tar.gzXCRP4BootromOFW-XC4-5.fallback.md5MinikernelMINIKERN_RBN64-xc4.p11.v7SEOSSEOS-mips-p1.tar.gz下载链接参见附录中的压缩包“6217P1 Hash”，FTP地址允许公网访问，下载完成后请进行MD5校验，以确保SEOS的可用性。为对应意外情况，建议新SEOS有多余1份的拷贝，现网SEOS文件也最好持有。2. 健康检查升级之前请对CE进行严格的健康检查，因升级过程中会涉及单台CE退网，检查之后方可确定是否能按单边方式进行。以下是两个Checklist示例，适于用不同的业务CE： 3. 工具和软件硬件：电脑笔记本，螺丝刀，网线，Console线软件：3CDaemon（或者其他的FTP/TFTP软件），UltraCompare，SecureCRT二、 升级步骤以下升级过程默认主XCRP为Slot 7，备XCRP为Slot 8。1. PC通过Console线接入SE路由器，并用网线与主XCRP网管口互联()，确保能相互ping通2. 保存配置文件,建议同时保存到原redback.cfg文件，外部MD和PC，并通过SecureCRT记录会话来存一份show configuration的txt或log文件，便于重启后做配置比对。# save configuration (敲入后请通过命令 dir 和 dir mate 确定redback.cfg & redback.bin文件的生成时间是否为刚刚操作的时间)# save configuration /md/20110915.cfg (cfg文件可以以日期命令，敲入后用 dir /md 进行确认)# save configuration t00/20110915.cfg (在PC上进行确认)3. 检查配置中是否存在 timeout session idle 命令，为避免使用PC传OS到XCRP的过程中出现会话中断，建议将其调为 timeout session idle 04. 单台SE退网，将其所有业务板卡全部 SHUT（通过show chassis中”H” Flag进行确认），并进行严格的拨打或上网测试(视CE业务而定)，确定业务正常后，再次保存配置，即save configuration5. 拔出备XCRP, 若有外置CF卡,为防止拔XCRP板时CF卡硬件损坏, 请先登录到备板上执行 “unmount /md”.localSE# telnet ( 对应XCRP7， 对应XCRP8)localstandby# show disk external detail ( mount on 字段为 /md )localstandby# unmount /md localstandby# show disk external detail ( mount on 字段为空 )注意：在拔出备XCRP前确保已拧紧相邻板卡的螺丝，若有人配合可帮助其按住相邻板卡，以免发生板卡连带现象。6. 将PC上开启FTP Server服务，将SEOS相关软件导入到SE中，执行升级操作并重启，这里可以分两种方式：（具体log参见附录）1) 将Bootrom/Minikernel/SEOS先导入到外部MD卡，再从MD卡作为数据源进行升级，涉及命令如下：(示例为XCRP4)# copy ftp:/lab:lab/OFW-XC4-5.fallback.md5 /md/OFW-XC4-5.fallback.md5# copy ftp:/lab:lab/MINIKERN_RBN64-xc4.p11.v7 /md/MINIKERN_RBN64-xc4.p11.v7# copy ftp:/lab:lab/SEOS-mips-p1.tar.gz /md/SEOS-mips-p1.tar.gz# dir /md# upgrade bootrom /md/OFW-XC4-5.fallback.md5 no-reload# upgrade minikernel /md/MINIKERN_RBN64-xc4.p11.v7 no-reload# release download /md/SEOS-mips-p1.tar.gz# show release# release upgrade2) 直接通过PC作为数据源进行升级，涉及命令如下：# upgrade bootrom ftp:/lab:lab/OFW-XC4-5.fallback.md5 no-reload# upgrade minikernel ftp:/lab:lab/MINIKERN_RBN64-xc4.p11.v7 no-reload# release download ftp:/lab:lab/SEOS-mips-p1.tar.gz# show release# release upgrade注意事项：1) 通过MD卡升级好处是可以避免SE升级进行时因网线接触问题产生数据错误或升级中断，缺点是占用MD卡容量（下载到MD卡前注意检查剩余空间），时间较长。若没有外置MD卡则不用考虑该方式；若通过MD卡release download不成功，可尝试直接从PC上release download。2) 文件名一定要写全，包括后缀（比如“tar”, “md5”等），否则会提示找不到该文件。3) Bootrom/Minikernel/SEOS中，SEOS一定要升，Bootrom/Minikernel需要将现网版本和目标版本进行比较，哪个不一致就升哪个，可通过show version来进行确认。建议升级顺序：Bootrom - Minikernel - SEOS (如果带no-reload关键字则bootrom/minikernel先后顺序不重要)，强烈要求升级bootrom/Minikernel时带有no-reload关键字，这样可以在最后一步升级SEOS时 (release upgrade) 一起重启，节省时间。4) 在XCRP3上从6143p3升级到6217p1，因6143p3 & 6145 XCRP3上的Bootrom/Minikernel版本要求与R6217p1相同，所以只升级SEOS即可,通过健康检查时show version进行确认。5) 在XCRP4上从6143p3/6145升级到6217p1，相关的 Bootrom/Minikernel/SEOS 均需要升级。7. SE重启后，通过show version / show release进行确认8. FPGA升级：通过show hardware detail检查业务板卡的FPGA版本是否匹配，若发现不匹配则通过命令 reload fpga x (x=slot number) 来进行升级。就目前经验来看，fege-60-2-port & 10ge-1-port均会在升到6217p1后出现FPGA不匹配的现象，整机重启时的log也会有相应的提示：Redback csm: FPGA mismatch detected on card fege-60-2-port in slot 1Redback csm: FPGA mismatch detected on card 10ge-1-port in slot 11最简单的检测FPGA是否需要升级的方法：逐个业务板卡尝试 reload fpga如果不需要升级系统会给出提示：localCE# reload fpga 6The reload command will upgrade the fpga code for the card in slot 6% Warning: operation will cause the card(s) to be re-initializedDo you really want to reload? (y/n) ySlot 6: no need to upgrade fpga (already up to date)如果需要升级则会继续执行：localCE# reload fpga 11The reload command will upgrade the fpga code for the card in slot 11% Warning: operation will cause the card(s) to be re-initializedDo you really want to reload? (y/n) yJul 19 01:33:54.062: %CSM-6-PORT: ethernet 11/1 link state DOWN, admin is DOWNJul 19 01:33:54.063: %CSM-6-PORT: ethernet 11/1 link state down, trigger source: C Configuration changed or hardware removedJul 19 01:33:54.221: %VLAN-6-INFO: port bvi bvi-VLAN1060 state DOWN admin is UPJul 19 01:33:54.227: %VLAN-6-INFO: port bvi bvi-VLAN1065 state DOWN admin is UPJul 19 01:33:54.229: %VLAN-6-INFO: port bvi bvi-VLAN1066 state DOWN admin is UPSlot 11: fpga upgrade in progress. Slot 11: the fpga upgrade process typically takes a few minutes.Slot 11: the card is not usable during the fpga upgrade.Slot 11: do not remove the card or shutdown the router during upgrade as it would corrupt the fpga. (后续省略)9. 确保所有业务板的FPGA板卡升级完毕后，再次整机重启（如果没有业务板卡需要升级FPGA则不必重启），重启之后记录一份show configuration，与之前的进行比较（忽略 timeout session 和板卡shutdown 产生的不同）10. 配置对比无异常后，插入备XCRP，进行同步，通过 show redundancy & show redundancy detail 确认同步进度11. 同步完成后，将所有业务板卡NO SHUT，测试业务是否正常，CE侧开始查看OSPF邻居，端口流量，路由和ARP表项等情况（参考升级前的Checklist），确保正常后save configuration，如果时间充裕可以通过telnet收集一份show tech12. 将另一台CE退网，进行严格的拨打或上网测试，确保业务正常后，执行同样的步骤进行升级13. 另一台CE升级完毕回归现网，进行最后的测试和该CE的健康检查，若客户要求或时间充裕，可再尝试单边确保该CE独立承载业务没有问题。三、 疑难问题解析正常升级中如果遇到：1) release upgrade 使CE重启后，启动过程中吐出大量log（如下所示），遇到该情况需耐心等待，不用操作，输出一段时间后会自动停止并向下进行。printf: not foundusage: rm -dfiPRrW file .printf: not foundmknod: node type must be b, c or p.printf: not foundusage: chmod -R -H | -L | -P -h mode file .2) SNMP配置丢失，此种情况适用于山东某些CE，原配置文件该部分显示为：snmp target IPNM-1 0 security-name Sdmcc_Ce version 2c view restricted snmp target IPNM-2 7 security-name Sdmcc_Ce version 2c view restricted snmp target IPNM-3 2 security-name Sdmcc_Ce version 2c view restricted snmp target IPNM-4 98 security-name Sdmcc_Ce version 2c view restricted升级后系统会将restricted识别为“Unknown view”，对于这样的CE，建议按规范增加对应配置：snmp target IPNM-1 0 security-name Sdmcc_Ce version 2c view Redback snmp target IPNM-2 7 security-name Sdmcc_Ce version 2c view Redback snmp target IPNM-3 2 security-name Sdmcc_Ce version 2c view Redback snmp target IPNM-4 98 security-name Sdmcc_Ce version 2c view Redback3) 在release download过程中SEOS无法导入Alternative Partition，伴有如下Log：%DLM-6-INFO: Restarting release sync server%DLM-6-INFO: Starting release sync server%DLM-3-GENERAL_ERR: Unable to bind to with errno 48%DLM-6-INFO: Release sync server stopped%DLM-6-INFO: Trying again in 30 seconds如果仅有如上DLM重启的信息，不会影响OS的导入，此现象概率性出现。-%DLM-3-GENERAL_ERR: Read from tar failed: Resource temporarily unavailable%DLM-3-GENERAL_ERR: tar process terminated due to receipt of signal 2%DLM-3-GENERAL_ERR: Agent failed to write to file 15: Broken pipe%DLM-3-GENERAL_ERR: Manager replies with error = 2, abort%DLM-3-GENERAL_ERR: download process terminated due to receipt of signal 2%DLM-3-GENERAL_ERR: tar process terminated due to receipt of signal 2%DLM-3-GENERAL_ERR: Download error, no installedOK file%DLM-3-GENERAL_ERR: Downloaded image does not appear to be valid, cleaning partition如果有以上信息出现，请尝试更换数据源，即从PC FTP直接导入或从MD卡导入。4) 备XCRP无法进行同步，请尝试如下操作：1 如果主控板可通过M2M Link识别备XCRP，可先重启DLM进程(process restart dlm)，没有效果的话可尝试 reload standby2 如果主控板长时间无法识别备XCRP，请用console口登录到备XCRP看是否已经进入OK模式，若进入OK模式，则需要在模式下输入 bootsys 进行备XCRP重启，通过console输出判定进入OK模式的原因，具体OK模式升级的方式请见后文。5) POS端口（oc48-1-port）无法UP，请尝试reload该板卡localCE# show system alarm allTimestamp Type Source Severity Description-Jul 19 01:42:24 xcrp 7 Major Backup fail: peer deadJul 19 01:42:24 xcrp 8 Critical Controller missingJul 19 01:42:34 xcrp 8 Major Controller auto switch completedJul 19 01:42:28 oc48-1-port 10/1 Minor Line remote defect indication (RDI-L) localCE# show conf port 10/1Building configuration.Current configuration:!card oc48-1-port 10 sonet-eu!port pos 10/1 description TO-SDWEF-BA-IPNET-RT02-GSR12416-POS11/0/0-2.5G link-dampening up 0 down 100 framing sdh alarm-report-only ais-p rdi-p no shutdown encapsulation ppp bind interface to-AR2_Media ChinaMobile_NGN_Media!endlocalCE# show port 10/1Slot/Port:Ch:SubCh Type State10/1 pos Down localCE# show port 10/1 detail pos 10/1 state is Down Line state : Down Admin state : UpLink Dampening : enabledLink up delay : 0Link down delay : 100Undampened line state : DownDampening Count : 0Media type : Sonet OC48 (SM)Encapsulation : pppMTU size : 4470 BytesLoopback : noneFraming : SDHSpeed : 2.48832 GbpsBandwidth : 2.39616 GbpsScramble : ONCrc : 32Tx C2 byte : 0x16Rx C2 byte : 0x16Line SF BER : 10E-4Line SD BER : 10E-7Report Only Alarms Path Alarms (report only): Path alarm indication signal (AIS-P) Path remote defect indication (RDI-P)Active Alarms : Line remote defect indication (RDI-L) Path Alarms : NONETx path-trace : 53445745 462d4e47 4e2d4345 31382d45 SDWEF-NGN-CE18-E 52494353 45383030 00000000 00000000 RICSE800. 00000000 00000000 00000000 00000000 . 00000000 00000000 00000000 00000d0a .Rx path-trace : 00000000 00000000 00000000 00000000 . 00000000 00000000 00000000 00000000 . 00000000 00000000 00000000 00000000 . 00000000 00000000 00000000 00000000 .localCE# reload card 10The reload command will restart the card in slot 10Do you really want to reload? (y/n) ySlot 10: card reMarking PTD channel DEAD for slot 10loaded successfullyJul 19 01:47:28.316: %CSM-6-PORT: pos 10/1 link state UP, admin is UPlocalCE# show port 10/1Slot/Port:Ch:SubCh Type State10/1 pos Up OK模式下的升级：OK模式下可以进行bootrom/minikernel/SEOS的升级，升级原则和普通模式一样，有需求的组件才升，OK模式主要在遇到突发情况下使用，基本命令如下：（适用于XC3 & XC4）Upgrade Bootrom:ok setenv ip-addr : /经验证该IP无法从PC去ping通，但可用ip-addr = :ok setenv server-ip-addr 00server-ip-addr = 00ok setenv gateway-ip-addr 00gateway-ip-addr = 00ok load net OFW-XC4-2.fallback.md5 OFW-XC4-2.fallback.md5 sending tftp request for file OFW-XC4-2.fallback.md5# / load net 命令默认通过 TFTP 传ok update-bootromProcessing bootrom image 0x8d000000, 736544 bytes. Dest: Primary:Checksum verification succeeded.Format and compatibility verification succeeded.Erasing bootrom.Programming bootrom.doneUpgrade Minikernel:ok load net MINIKERN_RBN64-xc4.p11.v7 MINIKERN_RBN64-xc4.p11.v7sending tftp request for file MINIKERN_RBN64-xc4.p11.v7#Checksum verification succeeded.ok write-kernelChecksum verification succeeded.Erasing Primary minikernel.Writing Primary minikernel 1578788 bytes.doneMinikernel Version: 11.7Upgrade SEOS:XC3:ok installsysEnter IP address : Enter netmask : Enter gateway IP address : Would you like to erase the vxWorks compact flash(Y/N) : yformating compact flash.doneWould you like to erase the BSD compact flash(Y/N) : yformatting compact flash.10+0 records inWould you like to erase the external disk(Y/N) : nWould you like to download a new release(Y/N) : yWould you like to install from external disk (E) or ftp (F): fInstall from FTPEnter IP address of server : Enter username : ericssonEnter path to release : SEOS-p1.tar.gzftping release.Password for ericsson: ftp:/ericsson/SEOS-p1.tar.gz (99037K)- # 99037K | 126.26K/s 15095K | 127.17K/s101413957 bytes transferred in 784.40 sec (126.26k/sec)copying releases to appropriate compact flash.Do you want to re-boot now (Y/N)? yrebooted by rootXC4:ok installsysEnter IP address : Enter netmask : Enter gateway IP address : Would you like to erase the BSD compact flash(Y/N): yFormatting compact flash.Would you like to erase the external disk(Y/N) : nWould you like to download a new release(Y/N) : yWould you like to install from external disk (E) or ftp (F): fInstall from FTPEnter IP address of FTP server : Enter username : aEnter path to release : /SEOS-mips-p1.tar.gzFtping release.Do you want to re-boot now (Y/N)? yrebooted by rootOK模式升级的注意事项：1) XCRP3对应的R6217p1版本不支持从OK模式（installsys）升级，会陷入无限重启，请勿执行此操作！2) 若备XCRP始终无法同步，而同步的内容包含Bootrom/Minikernel，可尝试在Console登录到备XCRP上，在OK模式下仅升级Bootrom/Minikernel再reboot该板卡，并观察同步情况。3) 在installsys下erase Vxworks & BSD SanDisk之后，该XCRP卡上存的配置文件会被一同删除，恢复出厂状态。4) 如果在升级过程中遇到Minikernel 校验错误，请尝试重新load net一次，如果还不行请更换一份Minikernel拷贝再试，log如下：ok load net M