CUDB和PG闰秒调整方案(2015.6.29)

1、 中国移动通信集团广东有限公司网络管理中心核网络维护室CUDB和PG闰秒调整方案中国移动广东公司网管维护中心核心网络维护室2015年6月 目录一、网络变更简报41.1网络变更背景41.2网络变更内容4二、变更风险点与业务影响评估52.1本次网络变更的主要风险点52.2本次变更实施人员52.3是否为首次变更52.4前期测试结果52.5业务影响评估52.5.1厂家意见52.5.2变更负责人意见52.6 综合影响评估62.6.1 计费影响62.6.2 综资及话务网管影响62.6.3 支撑系统调整62.6.4 性能统计影响62.6.5 业务流程影响62.7 领导决策点6三、操作内容与流程63.1 相关

2、拓扑图63.1 操作内容与流程7四、详细实施步骤84.1 变更前准备84.2 详细实施步骤9五、测试与验证方案105.1 验证方案（定期拨测）105.1 验证方案（定期指标）10六、应急方案116.1 本次变更可能出现的问题116.2 应急与倒回方案11七、BIBYSITTING监测要求13八、联系人表13九、备注14一、网络变更简报1.1网络变更背景本次操作是针对新增NGBOSS服务开通主机的静态路由配置的方案1.2网络变更内容修改网络配置 添加和NGBOSS相关的两条静态路由变更涉及网元网络变更内容网络影响情况变更操作时间业务影响时间业务受影响网元修改PG和CUDB的NTP配置无2015/

3、06/30 00:00 05:00无无二、变更风险点与业务影响评估2.1本次网络变更的主要风险点序号风险点具体描述变更负责人意见1局数据变更修改NTP服务器启动配置22.2本次变更实施人员2.3是否为首次变更是2.4前期测试结果修改配置后，观察设备的性能统计和告警是否正常2.5业务影响评估2.5.1厂家意见预计不会对网络业务产生不良影响。2.5.2变更负责人意见预计不会对网络业务产生不良影响。2.6 综合影响评估2.6.1 计费影响不会对计费产生影响变化。2.6.2 综资及话务网管影响相关网元信息在综资和话务网管中不需要更新。2.6.3 支撑系统调整不需要在支撑系统上补充配置，例如亿阳主动性能

4、监控、IPNET网管等。2.6.4 性能统计影响不会对网元或网管上的性能统计报表产生影响。2.6.5 业务流程影响不会对业务流程产生变化。2.7 领导决策点序号需要领导决策点具体描述变更负责人意见12三、操作内容与流程3.1 相关拓扑图3.1 操作内容与流程序号操作内容实施人实施时间结果确认1修改NTP服务器启动配置2015/06/30 00:00 2四、详细实施步骤4.1 变更前准备PG:序号变更准备项实施人配合人实施时间结果确认1设备厂家网元类型序号核查项核查及判定标准备注说明爱立信PG1dvemon status n allCAI3G ports should be E/E on all

5、 PL.MML ports should be E/E on all PL.CLI ports should be D/D on all PL.OAM ports should be E/E on all PL.Application server should be E/E on all PL.Deployment error number should be 0 on all PL.ESA processes should be UP on all the nodes.License server should be UP on both SC.Block proxy should be

6、UP on PL3.Reporter should be UP on all PL.Zookeeper should be UP on both SC and PL-3.EMA爱立信PG2fmacivealarmsThere should be no critical alarms.EMA2检查PG的proclog，确认各类指令的发放成功率指标检查，精确到指令级别（ADD_SUB、LST_SUB等）3备份所有GEP3板卡的NTP配置cp /etc/init.d/ntp /etc/init.d/ntp.bak4CUDB:序号变更准备项实施人配合人实施时间结果确认1设备厂家网元类型序号核查项核查及

7、判定标准备注说明爱立信CUDB1cudbSystemStatus(1) The time and date is right,(2)CUDB software version is right, (3)11B:System Monitor threads status on each system controller board are OK; 13B:System Monitor leader and BC Server status are OK.(4)Clusters status on each PL and DS is OK, the memory usage lower than

8、 80% (5) There are 4 NDBs process are running on PL Unit and 2 NDBs process are running on each DS Unit, (6) Replication Channels are OK, 'M': Master, 'Sn': Slave replication channel #n is active. (7)There is no new alarm is printed, more information refer to chapter 2.2.16 (5) MySQL

9、 servers connection are listed as “OK”; (6) CUDB Processes in OAMs, PLs and DSs are listed as “running”;爱立信CUDB2fmacivealarmsThere should be no critical alarms.23备份所有GEP3板卡的NTP配置cp /etc/sysconfig/ntp /etc/sysconfig/ntp.bak44.2 详细实施步骤步骤要求细化到指令级。PG:序号详细实施步骤（细化到指令级）实施人配合人实施时间结果确认1登录PG的第一块GEP32更改NTP工作模式

10、，修改ntp配置文件sed -i 's/-ng/-xng/g' /etc/init.d/ntp3然后重启板卡上的ntp服务service ntp restart4在其余板卡重复上述步骤CUDB:序号详细实施步骤（细化到指令级）实施人配合人实施时间结果确认1登录CUDB的备用SC2更改NTP工作模式，修改ntp配置文件/etc/sysconfig/ntp，将“NTPD_OPTIONS="-g -u ntp:ntp"”改为“NTPD_OPTIONS="-x -g -u ntp:ntp"sed -i 's/-g -u ntp:ntp /

11、-x -g -u ntp:ntp/g' /etc/sysconfig/ntp3然后重启板卡上的ntp服务rcntp restart4等2分钟5登录CUDB的主用SC6更改NTP工作模式，修改ntp配置文件/etc/sysconfig/ntp，将“NTPD_OPTIONS="-g -u ntp:ntp"”改为“NTPD_OPTIONS="-x -g -u ntp:ntp"sed -i 's/-g -u ntp:ntp /-x -g -u ntp:ntp/g' /etc/sysconfig/ntp7然后重启板卡上的ntp服务rcntp

12、 restart五、测试与验证方案5.1 验证方案（定期拨测）序号拨测内容网元验证频度验证结果验证人5.1 验证方案（定期指标）PG：序号指标网元验证频度验证结果验证人1检查PG的proclog，确认各类指令的发放成功率指标检查，精确到指令级别（ADD_SUB、LST_SUB等）cat /var/proclog/raw/proclog* | grep ADD_SUB | grep FAILED | wc -lcat /var/proclog/raw/proclog* | grep ADD_SUB | grep SUCC | wc lPG一次CUDB：序号指标网元验证频度验证结果验证人1检查CU

13、DB状态 cudbSystemStatusCUDB一次六、应急方案6.1 本次变更可能出现的问题系统有可能出现未知的软件错误，对业务发放造成影响。6.2 应急与倒回方案PG:使用备份的NTP配置文件倒回cp /etc/init.d/ntp.bak /etc/init.d/ntp然后重启NTP服务service ntp restartCUDB:使用备份的NTP配置文件倒回cp /etc/sysconfig/ntp.bak /etc/sysconfig/ntp然后重启NTP服务rcntp restart七、BIBYSITTING监测要求PG:设备厂家网元类型序号核查项核查及判定标准备注说明爱立信P

14、G1dvemon status n allCAI3G ports should be E/E on all PL.MML ports should be E/E on all PL.CLI ports should be D/D on all PL.OAM ports should be E/E on all PL.Application server should be E/E on all PL.Deployment error number should be 0 on all PL.ESA processes should be UP on all the nodes.License

15、server should be UP on both SC.Block proxy should be UP on PL3.Reporter should be UP on all PL.Zookeeper should be UP on both SC and PL-3.EMA爱立信PG2fmacivealarmsThere should be no critical alarms.EMA爱立信PG15cat /var/proclog/raw/proclog* | grep ADD_SUB | grep FAILED | wc -l确保所有指令的成功率都在正常范围EMAcat /var/p

16、roclog/raw/proclog* | grep ADD_SUB | grep SUCC | wc lCUDB:设备厂家网元类型序号核查项核查及判定标准备注说明爱立信CUDB1cudbSystemStatus(1) The time and date is right,(2)CUDB software version is right, (3)11B:System Monitor threads status on each system controller board are OK; 13B:System Monitor leader and BC Server status are

17、OK.(4)Clusters status on each PL and DS is OK, the memory usage lower than 80% (5) There are 4 NDBs process are running on PL Unit and 2 NDBs process are running on each DS Unit, (6) Replication Channels are OK, 'M': Master, 'Sn': Slave replication channel #n is active. (7)There is no new alarm is printed, more information refe