案例分析一次C4506交换机CPU利用率过高的处理过程

本文格式为Word版，下载可任意编辑——案例分析一次C4506交换机CPU利用率过高的处理过程案例分析：一次C4506交换机CPU利用率过高的处理过程刚处理过一次C4506交换机CPU利用率过高（1）

通过交换机showproccpu的命令输出的信息可以看到：

在5秒、1分钟、5分钟内Cat4kMgmtLoPri进程的CPU占用率分别为91.69%92.18%92.23%

显然是这个进程消耗了大量的CPU运行资源，从而导致交换机没有新的资源处理新的请求。

处理器运行两个进程，Cat4kMgmtHiPri和Cat4kMgmtLoPri，这两个进程同其他进程一样要占用cpu时间。当Cat4k平台上某项进程占用cpu超出了应分派的cpu时间，Cat4kMgmtLoPri会接纳这项进程，使其他进程能够得到cpu时间。同样当某个进程占用cpu时间没有超过规定的cpu利用率，Cat4kMgmtHiPri会接纳这个进程。

而Cat4kMgmtLoPri进程CPU占用率超过90%，我们判断一定是某一个进程大大超过了CPU应分派的时间，而Cat4kMgmtLoPri进程试图接纳这一进程，而导致了Cat4kMgmtLoPri进程的CPU占用时间超过了90%。根据这一思路只要我们找出这个进程，将其关闭或找出这个进程的作用，就可以解决这个问题并能够找出CPU占用率过高的原因。

showproccpu

CPUutilizationforfiveseconds:99%/0%;oneminute:99%;fiveminutes:99%

PIDRuntime(ms)InvokeduSecs5Sec1Min5MinTTYProcess101300.00%0.00%0.00%0ChunkManager

255221883250.00%0.00%0.00%0LoadMeter

316013112210.00%0.00%0.00%0SpanTreeHelper

40100.00%0.00%0.00%0DeferredEvents

51039241473270540.00%0.06%0.05%0Checkheaps

64220000.00%0.00%0.00%0PoolManager

70200.00%0.00%0.00%0Timers80200.00%0.00%0.00%0SerialBackgroun

90100.00%0.00%0.00%0AAA_SERVER_DEADT

100200.00%0.00%0.00%0AAAhigh-capacit

110100.00%0.00%0.00%0Policy

Manager

1216182580.00%0.00%0.00%0IPCDynamicCach

130100.00%0.00%0.00%0IPCZoneManager

1464410913550.00%0.00%0.00%0IPCPeriodicTim

1541210913630.00%0.00%0.00%0IPCDeferredPor

160100.00%0.00%0.00%0IPCSeatManager

170100.00%0.00%0.00%0IFSAgentManage

181624845140053160.23%0.21%0.22%0ARPInput

19681161810.00%0.00%0.00%0EntityMIBAPI

200100.00%0.00%0.00%0SERIALA'detect

2185610913670.00%0.00%0.00%0DynamicARPInsp

22206027333750.00%0.00%0.00%0HCCounterTimer

230100.00%0.00%0.00%0CriticalBkgnd

24171691507180.00%0.00%0.00%0NetBackground

250500.00%0.00%0.00%0Logger2671610911760.00%0.00%0.00%0TTYBackground

272072111013180.00%0.00%0.00%0Per-SecondJobs

28382921947196670.00%0.04%0.00%0Per-minuteJobs

29285850888073603244.63%3.81%3.72%0Cat4kMgmtHiPri

308437156467467994125091.69%92.18%92.23%0Cat4kMgmtLoPri

312604935850.00%0.00%0.00%0GaliosReschedul

320200.00%0.00%0.00%0Helper

330200.00%0.00%0.00%0NAMManager

IOSACL341522760000.00%0.00%0.00%0rftask3512428405853300.00%0.00%0.00%0NetInput

3611620230225040.07%0.03%0.00%0Computeloadavg

3708400.00%0.00%0.00%0BACKCHECK

380100.00%0.00%0.00%0chkptmessageha

390200.00%0.00%0.00%0cpf_process_msg_

400100.00%0.00%0.00%0cpf_process_ipcQ

4101400.00%0.00%0.00%0AggMgrProcess

420100.00%0.00%0.00%0SFF8472

4301500.00%0.00%0.00%0Collectionproce

440300.00%0.00%0.00%0CEFswitchingba

450200.00%0.00%0.00%0AAADictionaryR

460200.00%0.00%0.00%0AAAServer

470100.00%0.00%0.00%0AAAACCTProc

480100.00%0.00%0.00%0ACCTPeriodicPr

4968882413997074920.71%0.62%0.62%0SpanningTree

50439655767780.00%0.01%0.00%0DTPProtocol

51401094630.00%0.00%0.00%0Ethchnl

523744109502340.00%0.01%0.00%0UDLD5312934120.00%0.00%0.00%0DHCPSnooping

54201825100.00%0.00%0.00%0Port-Security

55252170049279115111.59%1.75%1.77%0IPInput

560100.00%0.00%0.00%0ICMPeventhandl

57919083791024240.07%0.14%0.11%0CDPProtocol

5816221720.00%0.02%0.00%0Exec611482534580.00%0.00%0.00%0CEFbackgroundp

620200.00%0.00%0.00%0XDRmcast

630100.00%0.00%0.00%0IPCLCMessageH

640100.00%0.00%0.00%0XDRRPPingBack

65892580.00%0.00%0.00%0XDRRPbackgroun

660100.00%0.00%0.00%0XDRRPTestBack

670100.00%0.00%0.00%068880641599285500.07%0.07%0.07%0CEF:IPv4proces

691843651110.00%0.00%0.00%0ADJbackground

70483671300.00%0.00%0.00%0L2MM711201447820.00%0.00%0.00%0MRD72106811946890.00%0.00%0.00%0IGMPSN730100.00%0.00%0.00%0IGMPSN-HA

740100.00%0.00%0.00%0SocketTimers

750200.00%0.00%0.00%0L2TRACESERVER

7628816798170.00%0.00%0.00%0TCPTimer

7740517840.00%0.00%0.00%0TCPProtocols

78101256411790.00%0.00%0.00%0HTTPCORE

790100.00%0.00%0.00%0CHKPTEXAMPLE

800100.00%0.00%0.00%0CHKPTDevTest

810200.00%0.00%0.00%0ATIP_UDP_TSK

820100.00%0.00%0.00%0DHCPSnoopingHA

830100.00%0.00%0.00%0Probe

IPIRDPInput

840100.00%0.00%0.00%0RARPInput

855380338081590.00%0.00%0.00%0DHCPDReceive

86113220795440.00%0.00%0.00%0IPBackground

8730018561610.00%0.00%0.00%0IPRIBUpdate

880100.00%0.00%0.00%0COPS89161371110.00%0.00%0.00%0ClusterL2

90761094660.00%0.00%0.00%0ClusterRARP

910200.00%0.00%0.00%0LOCALAAA

920200.00%0.00%0.00%0AAACachedServe

930200.00%0.00%0.00%0TPLUS940300.00%0.00%0.00%0RADIUSTESTCMD

950200.00%0.00%0.00%0AAASENDSTOPEV

9611256754707140.15%0.04%0.01%0PMCallback

9738030391250.00%0.00%0.00%0VLANManager

98891380.00%0.00%0.00%0DHCPDTimer

994162500.00%0.00%0.00%0VTPTrapProcess

1000200.00%0.00%0.00%0DHCPSecurityHe

1010100.00%0.00%0.00%0DiagCard1/-1

1020100.00%0.00%0.00%0DiagCard2/-1

1030100.00%0.00%0.00%0DiagCard3/-1

1040100.00%0.00%0.00%0DiagCard4/-1

1054124768650.00%0.00%0.00%0SyslogTraps

1060200.00%0.00%0.00%0VTPMIB

EDITBUFF

1070300.00%0.00%0.00%0SPANswitch

1080200.00%0.00%0.00%0SNMPTimers

1090200.00%0.00%0.00%0IPSNMP1100100.00%0.00%0.00%0PDUDISPATCHER

1110100.00%0.00%0.00%0SNMPENGINE

1120100.00%0.00%0.00%0SNMPConfCopyPro

113804916320.00%0.00%0.00%0SNMPTraps

1141344111625120.00%0.00%0.00%0NTP11532430976100.00%0.00%0.00%0DHCPDDatabase

1163565465360.00%0.00%0.00%0Systempolling

通过showplatformhealth命令进一步更为详细的检查Catalyst4506交换机的进程我们发现K2CpuManReview进程有异常，如下：

showplahea%CPU%CPURunTimeMaxPriorityAverage%CPUTotalTargetActualTargetActualFgBg5SecMinHourCPU

Lj-poll1.000.0121441005000000:26

GalChassisVp-review3.000.1710281005000002:50

S2w-JobEventSchedule10.000.251091005000004:35

Stub-JobEventSchedul10.002.36103410050022135:16

StatValueManUpdate1.000.05101005000001:58

Pim-review0.100.00101005000000:08

Ebm-host-review1.000.31841005000000:27

Ebm-port-review0.100.00101005000000:00

Protocol-aging-revie0.200.00201005000000:00

Acl-Flattener1.000.001051005000000:00

KxAclPathMancreate/1.000.001051005000000:14

KxAclPathManupdate2.000.0010141005000000:00

KxAclPathManreprogr1.000.00211005000000:00

TagMan-RecreateMtegR1.000.001051005000000:00

K2CpuManReview30.0069.843026100500112106781248:14

K2AccelPacketMan:Tx10.002.74200100500121210254:45

K2AccelPacketMan:Au0.100.00001005000000:00

K2AclMan-taggedFlatA1.000.001051005000000:00

K2AclCamManstaleen1.000.001051005000000:00

K2AclCamManhwstats3.000.501051005000007:24

K2AclCamMankxstats1.000.011051005000003:03

K2AclCamManAuditre1.007.761051005005004:07

K2AclPolicerTableMan1.000.001011005000000:17

K2L2AddressTableR2.003.251251005000007:25

K2L2NewStaticAddr2.000.001051005000000:00

K2L2NewMulticastA2.000.001051005000000:00

K2L2DynamicAddress2.000.001051005000000:00

K2L2VlanTableRevi2.000.001281005000000:02

K2L2DestinationCa2.000.001001005000000:00

K2PortManReview2.001.53151110050021126:42

Gigaport65535Review0.400.04411005000001:15

Gigaport65535Review0.400.07411005000001:15

Gigaport65535Review0.400.06411005000001:15

Gigaport65535Review0.400.05411005000001:17

Gigaport65535Review0.400.05411005000001:17

Gigaport65535Review0.400.06411005000001:16

Gigaport65535Review0.400.05411005000001:17

Gigaport65535Review0.400.05411005000001:17

Gigaport65535Review0.400.12401005000005:23

Gigaport65535Review0.400.09401005000005:20

Gigaport65535Review0.400.08401005000005:17

Gigaport65535Review0.400.09401005000005:19

Gigaport65535Review0.400.04411005000001:18

Gigaport65535Review0.400.08411005000001:17

Gigaport65535Review0.400.06411005000001:16

Gigaport65535Review0.400.06411005000001:16

Gigaport65535Review0.400.05411005000001:24

Gigaport65535Review0.400.05411005000001:22

Gigaport65535Review0.400.07451005000001:22

Gigaport65535Review0.400.07411005000001:22

Gigaport65535Review0.400.074111005000001:22

Gigaport65535Review0.400.06411005000001:23

Gigaport65535Review0.400.05491005000001:22

Gigaport65535Review0.400.05411005000001:21

Gigaport65535Review0.400.08401005000005:19

Gigaport65535Review0.400.11401005000005:22

Gigaport65535Review0.400.08401005000005:19

Gigaport65535Review0.400.10401005000005:19

Gigaport65535Review0.400.08401005000005:18

Gigaport65535Review0.400.09401005000005:18

Gigaport65535Review0.400.11401005000005:20

Gigaport65535Review0.400.09401005000005:19

K2Fibcamusagerevi2.000.001501005000000:00

K2FibIrmFibReview2.000.001501005000000:00

K2FibVrfDefaultRo2.000.001501005000000:00

K2FibAdjRepopRevie2.000.001501005000000:00

K2FibVrfUnpuntRev2.000.001501005000000:08

K2FibConsistencyCh1.0012.045210050021127:08

K2FibAdjManStatsRe2.000.381081005000005:20

K2FibAdjManHostMov2.000.001071005000000:14

K2FibAdjManAdjChan2.000.001001005000000:00

K2FibMulticastSigna2.000.031021005000000:46

K2FibMulticastEntry2.000.001061005000000:00

K2FibMulticastIrmM2.000.001071005000000:00

K2FibFastDropManRev2.000.00701005000000:00

K2FibPbrroutemapr2.000.412051005000004:02

K2FibPbrflataclpr2.000.082011005000000:45

K2FibPbrconsolidati2.000.011001005000000:12

K2FibPerVlanPuntMan2.000.001521005000000:00

K2FibFlowCacheflow2.000.001001005000000:07

K2FibFlowCacheflow2.000.001001005000000:00

K2FibFlowCacheadjr2.000.001001005000000:07

K2FibFlowCacheflow2.000.001001005000000:02

K2MetStatsManReview2.000.15521005000006:51

K2FibMulticastMETS2.000.001001005000000:00

K2QosDblManRateDBL2.000.09701005000001:48

IrmFibThrottlerThro2.000.07731005000000:43

K2VlanStatsManRevi2.000.9315410050011013:44

K2PacketMemoryDia2.000.3115810050021016:06

K2L2AgingTableRe2.000.072031005000002:42

RkiosPortManPortRe2.005.07123510050044357:49

RkiosModuleStateR4.000.024011005000000:30

RkiosOnlineDiagRe4.000.014001005000000:26

RkiosIpPbrIrmPortR2.000.011011005000000:40

RkiosAclManReview3.000.053011005000000:55

MatManReview0.500.00401005000000:00

Slot2ILCManagerR3.000.001001005000000:00

Slot2ILCS2wManRe3.000.001001005000000:00

Slot3ILCManagerR3.000.001001005000000:00

Slot3ILCS2wManRe3.000.001001005000000:00

Slot4ILCManagerR3.000.001001005000000:00

Slot4ILCS2wManRe3.000.001001005000000:00

Slot5ILCManagerR3.000.001001005000000:00

Slot5ILCS2wManRe3.000.001001005000000:00

Slot6ILCManagerR3.000.001001005000000:00

Slot6ILCS2wManRe3.000.001001005000000:00

EthHoleLinecardMan(12.000.031001005000000:29

EthHoleLinecardMan(22.000.181041005000003:55%CPUTotals211.80111.31

AllocationceilingCurrentallocationkbytes%inusekbytes%inuse

Linecard1'sStore258.004%12.04100%

Linecard2'sStore258.004%12.72100%

Linecard3'sStore258.0060%155.24100%

Linecard4'sStore258.0050%131.53100%

Linecard5'sStore258.000%0.000%

Linecard6'sStore258.000%0.00

0%

TSMobjectsRkiosSysPacketBuf250.000%0.560%

PacketBufRaw20355.00100%20355.00100%

PacketBufRawJumbo732.8125%183.20100%

Packet1026.562%30.1799%

PacketInfoItem390.620%0.190%

VbufNodes240080.750%0.000%

VbufNodes160055.750%12.190%

VbufNodes40073.008%10.2661%

VbufNodes6462.000%3.390%

GalGbicEntrys11.340%0.000%

PimPhyports875.0024%215.25100%

PimPorts796.8731%247.03100%

PimModules150.001%2.34100%

PimSlots5.002%0.11100%

PimChassis33.506%2.09100%

EbmVlans2688.001%30.18100%

EbmVlanGroupEntrys1920.000%4.80100%

EbmPorts184.0030%55.70100%

EbmPortHostEntrys384.000%0.000%

EbmIeNodes536.000%4.18100%

EbmPortVlanAclFeatur896.000%0.000%

EbmSortedHostTableIt1.870%0.000%

EbmSortedGroupTableI1.750%0.050%

IrmVrfs6.851%0.10100%

IrmFibAdjs768.001%16.1283%

IrmPortEtherAddrEntr500.000%0.000%

IrmFibEntries10240.000%39.7686%

AclL4Op384.000%0.17100%

AclL4OpTriplet256.000%0.15100%

AclClassifier768.000%0.18100%

AclFeature2512.680%0.42100%

Acl384.000%0.10100%

Ace1280.000%1.40100%

AceActionDescStorage256.000%0.000%

AclListNode256.000%0.06100%

AceListNode25600.000%0.5561%

AclClassifierActionL512.000%0.09100%

AclLayerFeatureListN512.000%0.0650%

AclClassifierListNod256.000%0.000%

TableMapManNameToTa27.000%0.000%

TableMapAllocator59.000%0.000%

FlatAcl512.000%0.0933%

FlatAce3840.000%3.9864%

FlatAceActionListNod76800.000%1.5964%

FlatAclL4OpSetStorag1024.000%0.000%

FlatAclCacheNode1024.000%0.06100%

QoSPolicers1672.000%0.000%

KxAclPath1024.000%2.25100%

KxAclPathListNode256.000%0.000%

ConfigToHwAfMap418.780%0.000%

HwToCfgAceMap192.000%0.000%

CommandTables48.0021%10.26100%

K2FibPbrFlatRouteMap515.850%0.000%

K2FibPbrExpandedFlat2304.000%0.000%

K2FibPbrFlatRouteMap320.000%0.000%

K2FibVrfs38.591%0.59100%

K2NetflowFPTEntri11.970%0.000%

K2TxPacket384.000%0.090%

K2TxPacketInfo256.000%0.170%

EbmVlanHostEntrys2048.001%35.5092%

MatEntrys7680.000%8.90100%

MatEntryTableIterato1.120%0.030%

RkiosAclManNamedGal43.750%0.000%

RkiosAclVlanMaps272.000%0.000%

RkiosAclVlanMapEnt1015.620%0.000%

RkiosQoSPolicyMaps1315.930%0.000%

RkiosQoSClassMaps896.000%0.02100%

AclToIosFilterMapLis384.000%0.000%

RkiosQoSPolicers380.000%0.000%

RkiosTableMapGalios3.000%0.000%

EventNodes84.000%0.758%

EventNodes84.002%17.7113%

KxAclTagPairNode2176.000%0.13100%

KxAclMappingTableEnt96.020%0.840%

KxAclMappingTableEnt64.010%0.000%

KxAclTaggedFlatAcl1024.000%0.06100%

RkisoIpPbrRouteMaps97.650%0.000%

IrmFlows256.000%0.000%

TSMtotals184649.7311%21299.0799%

IOS软件设计的K2CpuManReview进程CPU占用率最高为30%而实际却为69.84，大大超过了软件当时设计得阀值，而其他的进程都在软件设计的设定的范围之内，因此我们可以初步判断是K2CpuManReview进程出现了意外，致使CPU的管理进程Cat4kMgmtLoPri试图接纳这一进程，而最终导致交换机CPU占用率过高，导致资源的最终耗尽。从而影响了整个网络运行的稳定性。

通过Cisco提供的文档来看，K2CpuManReview进程主要处理“Hit〞CPU的数据包，Catalyist4506交换方式是CEF,并不是过程交换，在正常的状况下除设备管理、监测、路由计算外，数据包的转发主要由交换引擎来完成，并不需要CPU作过多的处理。K2CpuManReview占用的CPU资源过多，但各接口的流量并没有明显增大，显然不是正常的负载过重导致，一定是好多异常的数据包Hit了CPU,为了查明白CPU受到的异常数据包性质。我们通过命令：Monitorsession1soureinterfacecpu

Monitorsession1desinterfacegi3/8?

把CPU收到的数据包镜像到了gi3/8端口，然后在gi3/8端口接上天元龙马的

Slot4ILCS2wManRe3.000.001001005000000:00

Slot5ILCManagerR