防火墙运维与排错.ppt

防火墙运维与排错,chen gang ,note: cisco ios firewall is covered in sec-3000 (troubleshooting cisco ios security features) and will not be covered in this presentation,agenda,packet flow understanding the architecture failover troubleshooting case studies online resources best practices,packet flow,understanding the packet flow,to effectively troubleshoot a problem, one must first understand the packet path through the network attempt to isolate the problem down to a single device then perform a systematic walk of the packet path through the device to determine where the problem could be for problems relating to the cisco asa/pix/ fwsm, always determine the flow: src ip, dst ip, src port, dst port, and protocol determine the interfaces through which the flow passes,note: all firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both,accounting,example flow,flow src ip: src port: 11030 protocol: tcp dst ip: 5 dst port: 80 interfaces source: inside destination: outside,with the flow defined, examination of configuration issues boils down to just the two interfaces: inside and outside,eng,client: ,server: 5,inside,servers,dmz,partner,outside,hosting,packet flow,understanding the packet flow,once the device and flow have been identified, walk the path of the packet through the device the packet path through the firewall is illustrated in the next several slides for troubleshooting, pay careful attention to where the packet can be dropped in the decision-making process,packet processing flow diagram,the diagram below will be referenced on the following slides; it is shown here enlarged for reference,packet processing: ingress interface,packet arrives on ingress interface input counters incremented software input queue is an indicator of load no buffers indicates packet drops, typically due to bursty traffic,asa-5540# show interface gb-ethernet1 interface gb-ethernet1 “inside“ is up, line protocol is up hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 ip address , subnet mask mtu 1500 bytes, bw 1 gbit full duplex 5912749 packets input, 377701207 bytes, 0 no buffer received 29519 broadcasts, 0 runts, 0 giants 0 input errors, 0 crc, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0),ingress interface,packet processing: locate connection,check first for existing connection if connection exists, flow is matched; bypass acl check if no existing connection tcp non-syn packet, drop and log tcp syn or udp packet, pass to acl checks,established connection: asa-5540# show conn tcp out 5:80 in :11030 idle 0:00:04 bytes 1293 flags uio syslog because of no connection, and non-syn packet: asa-6-106015: deny tcp (no connection) from /11031 to 5/80 flags psh ack on interface inside,packet processing: acl check,first packet in flow is processed through interface acls acls are first match first packet in flow matches ace, incrementing hit count by one denied packets are dropped and logged,packet permitted by acl: asa-5540b# show access-list inside access-list inside line 10 permit ip any (hitcnt=1) syslog when packet is denied by acl: asa-4-106023: deny tcp src inside:/11034 dst outside:5/80 by access-group “inside“,packet processing: match translation,first packet in flow must match a translation rule* a quick route lookup is done only to determine egress interface translation rule can be to nat, or not to nat nat order of operations dictates what happens with overlapping translation rules once translation rule is matched, connection is created,translation exists: asa-5540# show xlate debug nat from inside: to outside:8 flags - idle 0:00:07 timeout 3:00:00 syslogs when no translation rule found: (305005no nat; 305006no global) asa-3-305005: no translation group found for tcp src inside:/11039 dst outside:5/80 asa-3-305006: regular translation creation failed for tcp src inside:/11040 dst outside:5/80,translation and nat order of operations,nat 0 access-list (nat-exempt) match existing xlates match static commands (cisco asa/pix first match; fwsm best match) static nat with and without access-list static pat with and without access-list match nat commands nat access-list (first match) nat (best match) if the id is 0, create an identity xlate use global pool for dynamic nat use global pool for dynamic pat,first match,packet processing: inspections/sec checks,inspections are applied to ensure protocol compliance (optional) customized aic inspections nat-embedded ips in payload additional security checks are applied to the packet (optional) packets passed to content security and control (csc) module,syslog from packets denied by security check: asa-4-406002: ftp port command different address: 1(1) to 30 on interface inside asa-4-405104: h225 message received from outside_address/outside_port to inside_address/inside_port before setup,inspections sec checks,question ?,what command will show you if packets are dropped by the inspection engines?,packet processing: nat ip header,translate the ip address in the ip header translate the port if performing pat update checksums (optional) following the above, pass packet to ips (aip) module,nat ip header,packet processing: egress interface,packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet) egress interface is determined first by translation rules if translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface example:,static (inside, outside) netmask static (dmz, outside) netmask ,inbound packets to get routed to inside based on order of statics,egress interface,packet processing: l3 route lookup,once on egress interface, an interface route lookup is performed only routes pointing out the egress interface are eligible remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface,syslog from packet on egress interface with no route pointing out interface: asa-6-110001: no route to 30 from ,packet processing: l2 address lookup,once a layer 3 route has been found, and next hop identified, layer 2 resolution is performed layer 2 rewrite of mac header if layer 2 resolution failsno syslog show arp will not display an entry for the l3 next hop debug arp will indicate if we are not receiving an arp reply,packet processing: transmit packet,packet is transmitted on wire interface counters will increment on interface output hardware and software queues indicate buffering at driver level, interface is busy,asa-5540# show interface gb-ethernet0 interface gb-ethernet0 “outside“ is up, line protocol is up hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c ip address 4, subnet mask mtu 1500 bytes, bw 1 gbit full duplex 3529518 packets input, 337798466 bytes, 0 no buffer received 32277 broadcasts, 0 runts, 0 giants 0 input errors, 0 crc, 0 frame, 0 overrun, 0 ignored, 0 abort 5585431 packets output, 359059032 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/2) software (0/0),xmit pkt,agenda,packet flow understanding the architecture failover troubleshooting case studies online resources best practices,cisco asa/pixunderstanding the architecture,cisco asa/pix platforms process all packets in software (via the central cpu) all packets are processed first in usually also first out no software limits on the number of aces (rules) that can be configured each ace takes a minimum of 212 bytes of ram cisco asa platforms have software imposed connection limits; cisco pix platforms do not (bound by ram) multi-cpu / multi-core systems hash packets in the same flow to the same cpu/core. 10 gig interfaces hash flow to same rx ring.,fwsmunderstanding the architecture,packets processed in hardware have zero impact on cpu similarly, if the cpu is pegged at 100%, this has zero impact on packets processed in hardware,fwsm process most packets in hardware, with some packets needing to be processed in softwarevia the control point (cp),fast path flow identification, security checks and nat in hardware,fwsm architectural overview,c6k backplane interface,session manager np 3,control point (cp) central cpu,fast path np 1,software,hardware,fwsm,control point acl compilation, fixups, syslog, aaa in software,session manager session establishment and teardown, aaa cache, acls,fast path np 2,fwsmacl rule limits,acl rules are about the only hardware limit users run into in multimode, acl resources are divided in 13 equal partitions (12 active, one backup) if you have less than 12 contexts, wasted reserved space,multi-context,single context,fwsmacl rule limits,fwsm 2.3 introduced resource acl-partitionset the number of acl partitions allocate-acl-partitionassigns a context to a specific partition fwsm 3.2 introduced resource-ruleallows further customization of a partition fwsm 4.0 introduced resource partitioncustomize the size of individual partitions access-list optimization enablemerges and/or deletes redundant and conflicting aces without affecting the policy,* see the appendix for examples of each of these commands,fwsmhardware limits,fwsm has several hardware limits that should be considered in your network design limits are hard set, but vary based on single or multimode some limits include:,increase over 2.3,increase over 3.1,*complete list in fwsm docs, appendix a (specifications),classifier in multimode,fwsm has a single mac address for all interfaces cisco asa/pix has single mac for shared interfaces (physical interfaces have unique macs) cisco asa/pix 7.2 introduces mac-address auto option to change this when the firewall receives a packet, it must classify it to determine where to send the packet packets are classified based on the following unique ingress interface/vlan packets destination ip matches a global ip,classifier in multimode,inbound traffic is classified to context ctx3, based on the global ip in the static,inside,,inside,,inside,,inbound packet,outside,vlan 4,vlan 5,vlan 6,fwsm,ctx1,ctx2,ctx3,msfc,.1,.2,.3,static (inside, outside) 9 ,shared interface,example,classifier in multimode,if the firewall is unable to classify a packet, the following syslog message is generated in the admin context*,%fwsm-6-106025: failed to determine security context for packet: vlan3 tcp src /1025 dest 5/80,*added to fwsm 3.1,agenda,packet flow understanding the architecture failover troubleshooting case studies online resources best practices,failover basics,active/standby vs. primary/secondary serial vs. lan failover stateful failover (optional) a failover only occurs when either firewall determines the standby firewall is healthier than the active firewall both firewalls swap mac and ip addresses when a failover occurs level 1 syslogs will give reason of failover,secondary (standby),primary (active),lan/serial,stateful,internet,corp,asa# show failover failover on failover unit primary failover lan interface: failover redundant5 (up) unit poll frequency 200 milliseconds, holdtime 1 seconds interface poll frequency 500 milliseconds, holdtime 5 seconds interface policy 1 monitored interfaces 2 of 250 maximum version: ours 8.1(2), mate 8.1(2)24 last failover at: 13:05:44 utc may 29 2009 this host: primary - active active time: 1366024 (sec) slot 0: asa5580 hw/sw rev (1.0/8.1(2) status (up sys) interface outside (41): normal interface inside (9): normal other host: secondary - standby ready active time: 0 (sec) slot 0: asa5580 hw/sw rev (1.0/8.1(2)24) status (up sys) interface outside (42): normal interface inside (0): normal stateful failover logical update statistics link : stateful redundant6 (up) stateful obj xmit xerr rcv rerr general 424525 0 424688 0 sys cmd 423182 0 423182 0,verifying failover configuration,interface monitoring,what triggers a failover?,power loss/reload (this includes crashes) on the active firewall ssm interface/module failure the standby becoming healthier than the active firewall,in lan based failover, what happens if the lan interface communication is severed?,what triggers a failover?,two consecutive hello messages missed on any monitored interface forces the interface into testing mode both units first verify the link status on the interface next, both units execute the following tests network activity test arp test broadcast ping test the first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failed,how well do you understand failover? what happens when,the lan interface communication is severed? you disable failover? (by issuing no failover) you rma/replace the primary unit? you dont define standby ips a member-interface in a redundant interface fails?,what to do after a failover,always check the syslogs to determine root cause example: switch port failed on inside interface of active firewall,asa-4-411002: line protocol on interface inside, changed state to down asa-1-105007: (primary) link status down on interface 1 asa-1-104002: (primary) switching to stndbyinterface check, mate is healthier,asa-1-104001: (secondary) switching to activemate want me active,syslogs from primary (active) firewall,syslogs from secondary (standby) firewall,asa# show failover state state last failure reason date/time this host - primary failed ifc failure 12:56:00 utc may 6 2009 inside: failed other host - secondary active none =configuration state= sync done =communication state= mac set,what to do after a failover,show failover state will provide specific details about the failure reason. this information is not saved across reboots,asa(config)# prompt hostname priority state asa/sec/act(config)#,other useful failover commands,failover exec mate allows you to execute commands on the peer and receive the response back. failover reload-standby only valid on active unit prompt changes the prompt to display failover priority and state.,agenda,packet flow understanding the architecture failover troubleshooting case studies online resources best practices,troubleshooting tools,syslogs debug commands show commands packet capture packet tracer,uses of syslogs,primary mechanism to record traffic to and through the firewall the best troubleshooting tool available,archival purposes,debugging purposes,syslog server,snmp server,console,trap .,syslog,asa syslog level vs. number of messages,fwsm syslog level vs. number of messages,what are modifiable syslog levels?,modifiable syslog levels allows one to move any syslog message to any level problem you want to record what exec commands are being executed on the firewall; syslog id 111009 records this information, but by default it is at level seven (debug) %asa-7-111009: user johndoe executed cmd: show run the problem is we dont want to log all 1715 other syslogs that are generated at debug level,no logging message level ,levels 0emergency 1alert 2critical 3errors 4warnings 5notifications 6informational 7debugging,how to create modifiable syslog levels,lower syslog message 111009 to level 3 (error) asa(config)# logging message 111009 level 3 now our syslog looks as follows %asa-3-111009: user johndoe executed cmd: show run to restore the default syslog level asa(config)# no logging message 111009 level 3,no logging message level ,solution,if you were only interested in logging one syslog message, how could you do it?,debug commands,debugs should not be the first choice to troubleshoot a problem debugs can negatively impact the cpu of the box, and also the performance of it; use with caution debugs are not conditional* know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug,* crypto conditional debugging was added to cisco asa/pix 8.0,debug icmp trace,valuable tool used to troubleshoot connectivity issues provides interface and translation information to quickly determine flow echo-replies must be explicitly permitted through acl, or icmp inspection must be enabled,redirecting debugs to syslog,problem log only debug output to syslog solution create a logging list with only syslog id 711001 enable debug output to syslogs log on the logging list,asa(config)# logging trap networkers .,asa(config)# logging list networkers message 711001 .,asa(config)# logging debug-trace info: logging debug-trace is enabled. all debug messages are currently being redirected to syslog:711001 and will not appear in any monitor session,show output filters,use output filters to filter the output of show command to only the information you want to see to use them, at the end of show , use the pipe character “|” followed by begin start displaying the output beginning at the first match of the regex, and continue to display the remaining output include display any line that matches the regex exclude display any line that does not match the regex grep same as include grep v same as exclude,show | begin|include|exclude|grep -v ,exam