IPRAN故障案例分析.docx

基站闪断故障案例分析摘要：某IPRAN站点在运行过程中出现频繁的设备闪断故障，造成此站点下挂4个3G站、3个4G站业务频繁闪断。背景2016年8月24日上午在网优人员给局方维护人员打来电话，说有数个站点出现频繁断站，然后过几分钟就好；我方工程师在收到局方告知后，立即与维护人员取得联系，为了不影响局方的业务以及考核指标，决定先将此站点下挂的站点端口进行关闭处理。问题描述及排查1. 登录到闪断的设备上，一般成环的节点很少会出现瞬断，但是不排除。首先检查瞬断节点的收发光功率；如果没问题就逐条检查环上的其他节点的光功率。一般情况下是光口实际收光超出临界值；zxr10#show perf-value interface gei_5/1laser-basis-cur(mA) : 6.8160laser-basis-min(mA) : 6.8160laser-basis-max(mA) : 6.8480laser-temp-cur(deg) : 39.8430laser-temp-min(deg) : 39.5620laser-temp-max(deg) : 39.8430optical-send-cur(dBm) : -6.1650optical-send-min(dBm) : -6.1670optical-send-max(dBm) : -6.1650optical-rcv-cur(dBm) : -7.5030optical-rcv-min(dBm) : -7.5100optical-rcv-max(dBm) : -7.5000send-bandwidth-usage : 0.0131rcv-bandwidth-usage : 0.0213zxr10#show perf-value interface gei_6/1laser-basis-cur(mA) : 17.7500laser-basis-min(mA) : 17.7500laser-basis-max(mA) : 17.8000laser-temp-cur(deg) : 33.2260laser-temp-min(deg) : 33.2260laser-temp-max(deg) : 33.2260optical-send-cur(dBm) : -5.9190optical-send-min(dBm) : -5.9190optical-send-max(dBm) : -5.8850optical-rcv-cur(dBm) : -7.2490optical-rcv-min(dBm) : -7.3680optical-rcv-max(dBm) : -7.2490send-bandwidth-usage : 0.0016rcv-bandwidth-usage : 0.00082. 检查设备温度，看风扇状态：设备温度过高会导致板卡假死，当温度降到板卡可以承受的温度的时候，板卡又会起来，所以就会出现闪断。zxr10#show versionZXR10 Router Operating System Software, ZTE CorporationZXR10 ROS Version V4.08.34ZXCTN 6220 Software, Version ZXCTN 620063006220 V2.10.00P02B10, RELEASE SOFTWARECopyright (c) 2001-2014 by ZTE CorporationCompiled Apr 24 2015, 18:46:39System image files are flash:System uptime is 244 day(s), 16 hour(s), 53 minute(s)SCCU, panel 1, masterMain Processor: PowerPC MPC8544 ProcessorSystem Memory : 1024M bytes, System FLASH : 128M bytesSystem NVRAM: 8K bytes, System Temperature: 41System Serial : 16383 , System BaudRate : 9600System Clock: PTP stratum 3Board Type: RSCCP2Serial Number: 269776601152, Part Number: 123353851003Main CPLD1 Version: V1.4, Main CPLD2 Version: V1.4CPU CPLD Version : N/A, PCI CPLD Version : N/AFPGA Version: V6.1 , Main PCB Number: 101001SCCU, panel 2, slaveMain Processor: PowerPC MPC8544 ProcessorSystem Memory : 1024M bytes , System FLASH: 128M bytesSystem NVRAM: 8K bytes, System Temperature: 43System Serial: 16383 , System BaudRate: 9600System Clock : PTP stratum 3Board Type : RSCCP2Serial Number: 269776600138, Part Number: 123353851003Main CPLD1 Version: V1.4, Main CPLD2 Version: V1.4CPU CPLD Version : N/A , PCI CPLD Version: N/AFPGA Version: V6.1 , Main PCB Number: 101001NPCI, panel 1Main Processor : W90N740 ProcessorSystem Memory: 16M bytes , System Temperature: N/ABoard Type: R4EGC , Logic Board Type : R4EGC_ETHPort Number : 4Serial Number : 269494602008, Part Number : 123352651015Main CPLD Version : V1.8FPGA Version : V0.8 , Main PCB Number : 081001NPCI, panel 2Main Processor : MPC8321 ProcessorSystem Memory: 128M bytes , System Temperature: 43Board Type : R16E1F, Logic Board Type : (TDM+IMA)Port Number : 16Serial Number : 269539101105 , Part Number: 123352651023Main CPLD Version : V2.3FPGA Version: V2.1 , Main PCB Number : 080902NPCI, panel 5Main Processor : W90N740 ProcessorSystem Memory: 16M bytes, System Temperature: 43Board Type: R4EGC , Logic Board Type : R4EGC_ETHPort Number: 4Serial Number: 269494601680, Part Number : 123352651015Main CPLD Version : V1.8FPGA Version : V0.8 , Main PCB Number : 081001NPCI, panel 6Main Processor : W90N740 ProcessorSystem Memory : 16M bytes , System Temperature: 43Board Type : R4EGC, Logic Board Type : R4EGC_ETHPort Number : 4Serial Number: 279887200696 , Part Number : 123352651015Main CPLD Version : V1.8FPGA Version : V0.8 , Main PCB Number : 081001zxr10#show fan-information PCB Version : 100801 Logical Version : V3.1 Fan1 : Normal Work Fan2 : Normal Work Fan3 : Normal Work Fan4 : Normal Work3. 查看端口是否有错包，当端口错包累计到一定程度后，很大可能性会出现设备闪断。zxr10#show interface gei_5/1gei_5/1 is up, line protocol is up, detect status is OK Byname is none Description is none The port is optical Duplex full Smb ipg 12 Smb mcc bandwidth 10 Mbits Mcc vlan id is 3024 Smb mcc priority 7 No loopback BW 1000000 Kbits Last clearing of show interface counters never 120 seconds input rate: 2428214 Bps, 8820 pps 120 seconds output rate: 1571473 Bps, 7147 pps Interface peak rate: input20166528Bpsoutput 8806412 Bps Interface utilization: input 1%, output 1% Input: Packets : 149523563988 Bytes: 43053642220163 Unicasts : 149493477935 Multicasts : 30086051 Broadcasts : 2 LittUndersize: 0 Oversize : 0 CRC-ERROR : 259107 Dropped : 2869615 Fragments : 0 Jabber : 0 MacRxErr : 0 Errframe : 0 Disframe : 2869615 UnicErrors : 0 MultiErrors : 0 BroErrors : 0 GoodJumb : 2591074607 64B : 0 65-127B : 44437087826 128-255B : 9959552083 256-511B : 85801245887 512-1023B: 746598738 1024-Max : 8579079454 AlignErrors : 0 Output: Packets : 138147568647 Bytes : 31382769692997 Unicasts : 138117545657 Multicasts : 30022986 Broadcasts : 4 Collisions : 0 LateCollision: 0 ColliTimes : 0 FlowContr : 0 DeExceFrame : 0 DecolliFra : 0 NonUniFrame : 30022990 ErrBytes : 0 ErrFrame : 0 MacControl : 0 JabberFrame : 0 GoodJumb : 167604028 LittUndersize: 0 64B : 0 65-127B : 39707412701 128-255B : 11607380287 256-511B : 85696141688 512-1023B : 532591611 1024-Max : 604042360 Oversize : 0zxr10#show interface gei_6/1gei_6/1 is up, line protocol is up, detect status is OK Byname is none Description is none The port is optical Duplex full Smb ipg 12 Smb mcc bandwidth 10 Mbits Mcc vlan id is 0 Smb mcc priority 7 No loopback BW 1000000 Kbits Last clearing of show interface counters never 120 seconds input rate: 83450 Bps, 809 pps 120 seconds output rate: 150818 Bps, 484 pps Interface peak rate : input 2781929 Bps, output 10989989 Bps Interface utilization: input 0%, output 0% Input: Packets :Bytes : 1259951803310 Unicasts :Multicasts : 30425156 Broadcasts : 2 LittUndersize: 0 Oversize : 0 CRC-ERROR :0 Dropped : 348 Fragments : 0 Jabber : 0 MacRxErr : 0 Errframe : 0 Disframe : 348 UnicErrors : 0 MultiErrors : 0 BroErrors : 0 GoodJumb : 7322682 64B : 21142804 65-127B :128-255B : 763326971 256-511B : 42375277 512-1023B: 33307440 1024-Max : 43759574 AlignE