网络故障诊断.ppt

1、Internetwork Troubleshooting,培训目标,掌握一般性故障判断与处理的流程 学会使用一至两种工具(软件)来进行故障诊断 学会使用系统提供的相关命令来进行网络故障诊断 了解一些常见的故障及其可能的原因,故障诊断流程,故障诊断流程,定义故障,收集现象,估计可能性,制订行动计划,实施行动,观察结果,重复过程,问题解决,结束,现象不存在,现象仍存在,故障诊断工具,故障诊断工具,仪表 万用表 电缆测试仪/光纤测试仪 误码表 数据发生器 协议分析仪,故障诊断工具,网管工具 HP OV Sun NetManager CA Unicenter TNG IBM Tivoli/NetVie

2、w CiscoWorks 其它工具,故障诊断工具,系统命令 Cisco路由器系统命令 show debug ping trace logging,故障诊断工具,系统命令 Solaris系统命令 ping ifconfig netstat nslookup arp 命令,故障诊断工具,其它工具 snoop sniffer tcpdump arpwatch 其它软件,路由器/交换机的故障诊断,正确理解路由器路由与交换的原理 了解Cisco路由器硬件结构 从路由器诊断命令的输出中发现问题 为Cisco技术支持人员提供诊断信息,Cisco路由器系统命令,Ping 命令(user),rtr1-c-1-b

3、jbjping Translating .domain server (202.97.16.195) OK Type escape sequence to abort. Sending 5, 100-byte ICMP Echoes to 192.18.97.195, timeout is 2 seconds: !.! Success rate is 80 percent (4/5), round-trip min/avg/max = 548/853/1204 ms,Ping 命令(user),Ping 扩展命令,r4#ping Protocol ip: Target IP address:

4、10.1.6.231 Repeat count 5: 10 Datagram size 100: 200 Timeout in seconds 2: 3 Extended commands n: y Source address or interface: 10.1.6.232 Type of service 0: 1 Set DF bit in IP header? no: Validate reply data? no: Data pattern 0 xABCD: 0 x0000 Loose, Strict, Record, Timestamp, Verbosenone: r Number

5、 of hops 9 : Loose, Strict, Record, Timestamp, VerboseRV: Sweep range of sizes n: Type escape sequence to abort. Sending 10, 200-byte ICMP Echos to 10.1.6.231, timeout is 3 seconds: Packet has data pattern 0 x0000 Packet has IP options: Total option bytes= 39, padded length=40 Record route: 0.0.0.0

6、0.0.0.0 0.0.0.0 0.0.0.0,Ping扩展命令(cont),Reply to request 0 (4 ms). Received packet has options Total option bytes= 40, padded length=40 Record route: 10.1.6.232 10.1.6.231 10.1.6.232 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 End of list Reply to request 1 (4 ms). Received packet has options Tot

7、al option bytes= 40, padded length=40 Record route: 10.1.6.232 10.1.6.231 10.1.6.232 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 End of list Reply to request 2 (4 ms). Received packet has options Total option bytes= 40, padded length=40 Record route: 10.1.6.232 10.1.6.231 10.1.6.232 0.0.0.0 0.0.

8、0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 End of list Success rate is 100 percent (10/10), round-trip min/avg/max = 1/4/8 ms,Trace命令(user),rtr1-c-1-bjbjtrace 202.97.18.60 Type escape sequence to abort. Tracing the route to flow- (202.97.18.60) 1 1p1-rtr3-c-1- (202.97.9.2) 0 msec 0 msec 0 msec 2 202.97.9.1

9、66 20 msec 16 msec 16 msec 3 2p0-rtr4-c-1- (202.97.9.82) 20 msec 16 msec 20 msec 4 6p0 x0-rtr2-r-1- (202.97.10.42) 20 msec 20 msec 20 msec 5 flow- (202.97.18.60) 20 msec * 16 msec,Trace 命令(user),Trace扩展命令,r4#trace Protocol ip: Target IP address: 10.1.6.231 Source address: 10.1.6.232 Numeric display

10、n: 5 Timeout in seconds 3: Probe count 3: Minimum Time to Live 1: 2 Maximum Time to Live 30: Port Number 33434: Loose, Strict, Record, Timestamp, Verbosenone: r Number of hops 9 : Loose, Strict, Record, Timestamp, VerboseRV: Type escape sequence to abort.,Trace扩展命令( Cont),Tracing the route to 10.1.6

11、.231 2 10.1.6.231 0 msec Received packet has options Total option bytes= 40, padded length=40 Record route: 10.1.6.232 10.1.6.231 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 End of list * 0 msec Received packet has options Total option bytes= 40, padded length=40 Record route: 10.1.6.232

12、 10.1.6.231 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 End of list,Show 命令,rtr1-c-1-bjbjsh int f4/0/0 FastEthernet4/0/0 is administratively down, line protocol is down Hardware is cyBus FastEthernet Interface, address is 0050.a253.8080 (bia 0050.a253.8080) Description: 100M Internet add

13、ress is 202.97.8.253/30 MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, rely 255/255, load 1/255 Encapsulation ARPA, loopback not set, keepalive set (10 sec), fdx, 100BaseTX/FX ARP type: ARPA, ARP Timeout 04:00:00 Last input 10w6d, output 10w6d, output hang never Last clearing of show interface counte

14、rs 18w4d Queueing strategy: fifo Output queue 0/40, 535 drops; input queue 0/75, 1716 drops 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec,Show命令(Cont),15628691 packets input, 2423133689 bytes, 0 no buffer Received 698983 broadcasts, 0 runts, 0 giants 0

15、input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 watchdog, 0 multicast 0 input packets with dribble condition detected 25793600 packets output, 4210267836 bytes, 0 underruns 0 output errors, 0 collisions, 3 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no ca

16、rrier 0 output buffers copied, 0 interrupts, 0 failures,Show 命令,rtr1-c-1-bjbjsh int s0/0/0 Serial0/0/0 is up, line protocol is up Hardware is cyBus Serial Description: to MCI (BEIJING-SACRAM 2048K003) Internet address is 204.189.216.146/30 MTU 1500 bytes, BW 1984 Kbit, DLY 20000 usec, rely 255/255,

17、load 1/255 Encapsulation HDLC, loopback not set, keepalive set (10 sec) Last input 00:00:07, output 00:00:07, output hang never Last clearing of show interface counters 18w4d Queueing strategy: fifo Output queue 0/40, 455018414 drops; input queue 0/75, 124 drops 5 minute input rate 0 bits/sec, 0 pac

18、kets/sec 5 minute output rate 15000 bits/sec, 1 packets/sec 110103206 packets input, 3264989278 bytes, 1 no buffer Received 0 broadcasts, 0 runts, 0 giants 11447 input errors, 9191 CRC, 0 frame, 28 overrun, 0 ignored, 2228 abort 3720562109 packets output, 503074661 bytes, 0 underruns 0 output errors

19、, 0 collisions, 534 interface resets 0 output buffers copied, 0 interrupts, 0 failures 214 carrier transitions 6921 alarm indications, 6959 remote alarms, 7003 rx LOF, 42 rx LOS DCD up, BER inactive, NELR inactive, FELR inactive,Show命令,rtr1-c-1-bjbjsh proc cpu CPU utilization for five seconds: 6%/5%

20、; one minute: 11%; five minutes: 9% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 262716720 287395415 914 0.57% 0.45% 0.47% 0 BGP Router 2 305835744 3100720 98635 0.00% 0.91% 1.16% 0 Check heaps 3 54720 408801 133 0.00% 0.00% 0.00% 0 Pool Manager 4 0 2 0 0.00% 0.00% 0.00% 0 Timers 5 0 1

21、 0 0.00% 0.00% 0.00% 0 OIR Handler 6 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager 7 0 1 0 0.00% 0.00% 0.00% 0 IPC Realm Manager 8 5196 2204477 2 0.00% 0.00% 0.00% 0 IPC Seat Manager 9 3672 687198 5 0.00% 0.00% 0.00% 0 ARP Input 10 0 1 0 0.00% 0.00% 0.00% 0 SERIAL Adetect 11 0 1 0 0.00% 0.00% 0.00% 0 M

22、icrocode Loader 12 0 1 0 0.00% 0.00% 0.00% 0 ATM ILMI Input,Show命令,rtr1-c-1-bjbjsh version Cisco Internetwork Operating System Software IOS (tm) GS Software (RSP-JV-M), Version 11.1(19)CC1 rtr1-c-1-bjbj uptime is 1 year, 14 weeks, 2 days, 16 hours, 0 minutes System restarted by reload at 01:23:52 PR

23、C Sun Jan 31 1999 System image file is slot0:rsp-jv-mz.111-19.CC1, booted via slot0 cisco RSP4 (R5000) processor with 262144K/2072K bytes of memory. R5000 processor, Implementation 35, Revision 2.1 (512KB Level 2 Cache) Last reset from power-on Chassis Interface. 5 VIP2 R5K controllers (2 FastEthern

24、et)(9 Serial)(2 HSSI)(2 POS). 2 FastEthernet/IEEE 802.3 interfaces. 9 Serial network interfaces. 2 HSSI network interfaces. 2 Packet over Sonet network interfaces. Configuration register is 0 x102,Debug 命令,r4#debug arp ARP packet debugging is on r4# 0:33:08: IP ARP req filtered src 0.0.0.0 0800.209f

25、.92b9, dst 198.32.64.12 ffff.ffff.ffff martian source 0:33:09: IP ARP req filtered src 0.0.0.0 0800.209f.92b9, dst 192.203.230.10 ffff.ffff.ffff martian source 0:33:09: IP ARP req filtered src 0.0.0.0 0800.209f.92b9, dst 128.8.10.90 f fff.ffff.ffff martian source 0:33:09: IP ARP req filtered src 0.0

26、.0.0 0800.209f.92b9, dst 198.32.64.12 ffff.ffff.ffff martian source 0:33:09: IP ARP: rcvd req src 10.1.5.168 0000.21d9.5722, dst 10.1.1.16 Ethernet0 0:33:09: IP ARP req filtered src 0.0.0.0 0800.209f.92b9, dst 192.203.230.10 ffff.ffff.ffff martian source 0:33:09: IP ARP req filtered src 0.0.0.0 0800

27、.209f.92b9, dst 202.12.27.33 ffff.ffff.ffff martian source 0:33:09: IP ARP req filtered src 0.0.0.0 0800.209f.92b9, dst 192.33.4.12 ffff.ffff.ffff martian source,Debug命令,r4#debug ip packet IP packet debugging is on r4# 0:36:49: IP: s=10.1.9.21 (Ethernet0), d=10.1.255.255, len 118, rcvd 0 0:36:49: IP

28、: s=10.1.5.94 (Ethernet0), d=10.1.255.255 (Ethernet0), len 234, rcvd 3 0:36:49: IP: s=10.1.9.189 (Ethernet0), d=255.255.255.255, len 140, rcvd 0 0:36:49: IP: s=10.1.9.189 (Ethernet0), d=255.255.255.255, len 140, rcvd 0 0:36:49: IP: s=10.1.8.236 (Ethernet0), d=255.255.255.255, len 138, rcvd 0 0:36:49

29、: IP: s=10.1.8.228 (Ethernet0), d=255.255.255.255, len 140, rcvd 0 0:36:49: IP: s=10.1.9.21 (Ethernet0), d=10.1.255.255, len 118, rcvd 0 0:36:49: IP: s=10.1.8.234 (Ethernet0), d=255.255.255.255, len 146, rcvd 0 0:36:49: IP: s=10.1.1.20 (Ethernet0), d=10.1.255.255 (Ethernet0), len 231, rcvd 3 0:36:49

30、: IP: s=10.1.7.7 (Ethernet0), d=10.1.255.255, len 146, rcvd 0 0:36:49: IP: s=10.1.9.8 (Ethernet0), d=10.1.9.255 (Ethernet0), len 231, redirected 0:36:49: IP: s=10.1.9.8 (Ethernet0), d=10.1.9.255 (Ethernet0), g=10.1.9.255, len 231, forward,Logging命令,Logging on 四种方式 console-logging console level buffe

31、r-logging buffered unix syslog logging ip-address logging trap level monitor-logging monitor level,UNIX Syslog主机,Syslog config file /etc/syslog.conf For example: local7.warn /var/log/local7.warn(make sure “tab” between local7.warn and /var/log/local7.warn, not space) The file local7.warn must exist

32、and be writable syslogd must be restarted by root ps -ef|grep syslogd kill -9 syslogd -d,Error message logging,Console(资源消耗最大),Terminal monitor,syslogd,buffer,Syslog 优先级,Syslog 文件,Jan 9 15:17:39 sendmail1138: NOQUEUE: 4: fl=0 x1, mode=206 66: CHR: size=0 Jan 9 15:17:39 sendmail1138: PAA01138: from=w

33、angjk, size=16 0, class=0, pri=30160, nrcpts=1, msgid= , relay=wangjklocalhost Jan 9 15:17:40 sendmail1140: PAA01138: to=root, ctladdr=wan gjk (237/200), delay=00:00:01, xdelay=00:00:01, mailer=local, stat=Sent Jan 14 08:49:41 sendmail225: starting daemon (8.9.3): SMTP+q ueueing00:15:00 Jan 15 07:20

34、:02 sendmail457: NOQUEUE: Null connection from l ocalhost 127.0.0.1 Jan 17 04:01:41 sendmail216: starting daemon (8.9.3): SMTP+q ueueing00:15:00,Solaris 系统命令,Ping命令,Ping 命令结果 unknown host-不能解析远程主机的域名 network unreachable-本地主机没有到远程主机的相应的路由 no answer-没有得到远程主机的响应 host is alive-得到远程主机的响应,traceroute,Time-

35、to-live值 由1开始 递增值为1 非法的UDP端口 默认值为33343,traceroute,Source,1st Gateway,2nd Gateway,Destination,ttl=1,decrease to 0 return error,decrease to 0 return error,decrease to 1 forward,decrease to 1 forward,decrease to 2 forward,reach destination port unreachable return error,ttl=2,ttl=3,Ifconfig命令,/disk1/sha

36、o% ifconfig -a lo0: flags=849 mtu 8232 inet 127.0.0.1 netmask ff000000 hme0: flags=863 mtu 1500 inet 10.252.5.111 netmask ffff0000 broadcast 10.252.255.255,Nslookup 命令,/disk1/shao% nslookup Default Server: Address: 10.1.1.7 Server: Address: 10.1.1.7 Name: Address: 209.10.217.9,Netstat命令,/disk3/shao%

37、 netstat -rn Routing Table: Destination Gateway Flags Ref Use Interface - - - - - - 209.151.24.44 10.1.1.2 UGHD 0 1 10.1.0.0 10.1.6.231 U 3 593 le0 224.0.0.0 10.1.6.231 U 3 0 le0 default 10.1.1.1 UG 0 37 127.0.0.1 127.0.0.1 UH 0 0 lo0,Netstat命令,/disk3/shao% netstat -i Name Mtu Net/Dest Address Ipkts

38、 Ierrs Opkts Oerrs Collis Queue lo0 8232 loopback localhost 42 0 42 0 0 0 le0 1500 tech 8134310 4 13963 1 25 0,Arp命令,/disk3/shao% arp -a Net to Media Table Device IP Address Mask Flags Phys Addr - - - - - le0 proxy 255.255.255.255 08:00:20:7a:fb:82 le0 10.1.1.2 255.255.255.255 00:a0:c9:8a:a8:f9 le0

39、ai-nas 255.255.255.255 00:b0:64:e9:d8:41 le0 10.1.7.7 255.255.255.255 08:00:20:b0:85:18 le0 aints1 255.255.255.255 00:a0:c9:8d:d9:95 le0 10.1.6.36 255.255.255.255 00:d0:59:08:28:50 le0 10.1.1.50 255.255.255.255 00:00:21:00:92:98 le0 aints3 255.255.255.255 00:90:27:c3:bb:d5 le0 0.0.0.0 255.255.255.25

40、5 08:00:20:9f:92:b9,厂家MAC地址列表,一般性故障诊断,Router Crash,What is crash? System has detected an unrecovered error and restarted itself What to collect? Show tech Show log Show environment all Do not reload router before getting all necessary information,Router Hang,What is router hang? System encountered a

41、 catastrophic failure but cant recover itself What to do next? Check if traffic can still flow through router If yes, disconnect all interfaces to see if console is available; If console is available, do show tech, show log, show environ all; if not, press break key to rommon mode,Router Hang(cont.)

42、,If no traffic can get through, press crtl-break key to rommon mode In rommon mode, type the following commands and get all the output K, context, cont and waiting for a while Do the same: K, context, cont Repeat above for several times （Break must be enabled in config-register for breaking to rommo

43、n config-register 0 x2002 reload to take effect ）,Memory Problems,Memory fragmentation memory may get fragmented into very small blocks so that all processes that require a larger block cannot get memory of the block size that it needs. Memory leak memory leak means that a particular process is not

44、freeing up any allocated memory when this process is done with it. This process keeps the RAM in use even when it doesnt need it. Lack of Memory,局域网故障诊断,有关的路由器命令 ping 及trace 命令 show arp show interface show ip route show ip access-list debug,局域网故障诊断,有关Solaris系统命令 arp命令 ping/traceroute命令 ifconfig命令 ne

45、tstat命令 nslookup命令,局域网故障诊断流程,本地主机配置正确?,能否ping本地路由器?,所有路由器是否可达?,远端主机配置正确?,N,Y,Y,Y,N,N,N,修改本地主机配置 (address,mask, default gateway),修改本地路由器配置 及局域网问题,修改所有路由器配置 及广域网问题,修改远端主机配置,现象与故障判断,现象与故障判断(Cont),广域网故障诊断,广域网故障诊断,相关的路由器命令 show interface serial show line show controller ping/traceroute debug,路由协议故障诊断,一般性

46、命令 show ip protocol show ip route sunmmary OSPF路由协议 show ip ospf database show ip ospf interface show ip ospf neighbor,BGP路由协议 show ip bgp show ip bgp summary show ip bgp community IS-IS路由协议 show clns neighbor show clns route show clns is-neighbor show isis database,串行链路环测,Debug 命令,Debug 命令 debug se

47、rial interface debug ppp negotiation debug serial packet 慎重地使用debug 命令!,Debug 命令,路由协议诊断,rtr1-c-1-bjbj sh ip route sum Route Source Networks Subnets Overhead Memory (bytes) connected 0 8 632 1088 static 0 1155 169772 157080 isis 4 415 28728 56984 Level 1: 0 Level 2: 419 bgp 64999 72757 12937 4798864

48、11842064 External: 1463 Internal: 84113 Local: 118 internal 2121 288456 Total 74882 14515 4997996 12345672,路由协议诊断-OSPF,r1-c-bjsh ip ospf database OSPF Router with ID (211.94.38.12) (Process ID 165) Router Link States (Area 0) Link ID ADV Router Age Seq# Checksum Link count211.94.38.7 211.94.38.7 841

49、 0 x800032DB 0 x38E5 14211.94.38.9 211.94.38.9 1421 0 x800017AC 0 x7AFF 5211.94.38.10 211.94.38.10 384 0 x8000155F 0 x6F25 3211.94.38.11 211.94.38.11 1725 0 x8000066A 0 xB478 3211.94.38.12 211.94.38.12 1558 0 x80002216 0 xCD0E 55211.94.38.13 211.94.38.13 1092 0 x80000CBB 0 xAFB3 2211.94.38.14 211.94

50、.38.14 1499 0 x800049C9 0 x7D75 43211.94.38.15 211.94.38.15 850 0 x800010A8 0 xDE95 8211.94.38.16 211.94.38.16 1450 0 x80003776 0 x228E 38211.94.38.17 211.94.38.17 764 0 x80000215 0 x43BD 18211.94.38.18 211.94.38.18 463 0 x800031F6 0 x1E32 21,路由协议诊断-OSPF,r1-c-bjsh ip ospf int FastEthernet0/1/0 is up

51、, line protocol is up Internet Address 211.94.39.2/29, Area 0 Process ID 165, Router ID 211.94.38.12, Network Type BROADCAST, Cost: 1 Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) 211.94.38.12, Interface address 211.94.39.2 Backup Designated router (ID) 211.94.38.11, Interface

52、 address 211.94.39.1 Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:09 Index 2/2, flood queue length 0 Next 0 x0(0)/0 x0(0) Last flood scan length is 3, maximum is 40 Last flood scan time is 0 msec, maximum is 4 msec Neighbor Count is 1, Adjacent neighbor cou

53、nt is 1 Adjacent with neighbor 211.94.38.11 (Backup Designated Router) Suppress hello for 0 neighbor(s),路由协议诊断-OSPF,r1-c-bjsh ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface211.94.38.11 1 FULL/BDR 00:00:35 211.94.39.1 FastEthernet0/1/0211.94.38.14 1 FULL/ - 00:00:35 211.94.40.14 A

54、TM4/0/0.1211.94.38.17 1 FULL/ - 00:00:37 211.94.40.22 ATM4/0/0.2211.94.38.20 1 FULL/ - 00:00:36 211.94.40.34 ATM4/0/0.3211.94.38.21 1 FULL/ - 00:00:37 211.94.42.2 ATM4/0/0.4211.94.38.24 1 FULL/ - 00:00:36 211.94.42.6 ATM4/0/0.5211.94.38.38 1 FULL/ - 00:00:38 211.94.42.10 ATM4/0/0.6211.94.38.46 1 FUL

55、L/ - 00:00:34 211.94.43.66 ATM4/0/0.7211.94.38.26 1 FULL/ - 00:00:34 211.94.43.70 ATM4/0/0.8211.94.38.27 1 FULL/ - 00:00:36 211.94.43.74 ATM4/0/0.9211.94.38.22 1 FULL/ - 00:00:34 211.94.43.90 ATM4/0/0.10211.94.38.40 1 FULL/ - 00:00:34 211.94.43.94 ATM4/0/0.11,路由协议诊断-BGP,rtr1-c-1-bjbjsh ip bgp BGP ta

56、ble version is 41226682, local router ID is 202.97.8.1 Status codes: s suppressed, d damped, h history, * valid, best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path * 3.0.0.0 206.253.223.2 50 0 6993 701 80 i * 204.189.216.145 50 0 3561 701 80

57、i * i 202.0.170.2 100 0 14923 701 80 i *i 202.0.170.2 100 0 14923 701 80 i * 4.0.0.0 206.253.223.2 50 0 6993 1 i * 204.189.216.145 50 0 3561 1 i *i 205.174.74.169 100 0 5727 1 i,路由协议诊断命令-BGP,rtr1-c-1-bjbjsh ip bgp sum BGP table version is 41225605, main routing table version 41225605 85846 network entries (500146/591202 paths) using 27630064