防火墙网关故障排除.ppt

UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,最佳实践:高级网关TroubleshootingT2,2,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,有效的Troubleshooting,坦白来讲，它可以作为一种艺术但是这不是一个忽略最佳实践的借口在做Troubleshoot一个问题时，一个小策划和应尽的努力是必要的。让我们从基本的开始，并且不断寻找最后解决所有问题.事后将很容易看到哪里出了错.,3,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,如果你真正troubleshooting一个问题，很可能你将寻求support的援助。首先需要你收集相关信息这能帮助支持工程师很快的定位问题。Question谁最有能力解决自己的问题？(意思是，取决于您收集的信息将最能解决问题),4,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Question谁最有能力帮助您解决问题?,5,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,因此，灾难发生后，现在该怎么做?,发生了什么变化?第一个冲动是去四处打听，看看已经做了什么，然后开始把此前更改还原，然后忽略这种冲动。,实际上.只是去校验.,6,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,回到基础来(从OSI底层向OSI高层的检查),7,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,线缆是否插好?(物理层检查),检查电源连接,网络线缆连接,8,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,网络是否正常?(速率与双工设置，接口up),8,9,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,下一步的检查(数据链路层检查arp、路由),9,10,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,检查应用的通信(传输层检查，源、目标地址),11,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,检查应用的通信(会话层检查抓包，查看会话流量),12,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,寻找应用通信信息,12,13,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,这就是事实.,验证到上层的连接事情开始变得火热,14,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,IPS的执法?(SmartDefense),15,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,战争模式，做好准备.,16,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,检查Firewall状态,16,17,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,系统负载问题?,17,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,了解防火墙内存,FW内存管理是复杂的.“fwctlpstat”是主要的分析工具.复杂的输出.许多方面产生误解.该系统具备自己的统计数据.Dontdistinguishfwfromtherest.Dontalwaysdistinguishnon-swappablememoryfromuserprocessmemory.(并不总从用户进程内存区分非可交换内存)结果需要理解它是如何工作.单独介绍,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,我是否有内存问题?,聚焦在防火墙内核内存的问题.(fwkernel即防火墙内核)如果是其他应用吃掉内存-那不是防火墙的问题.如果是守护进程吃掉内存-那不是内核的问题.fwctlpstat是个关键.所有的fwkernel分配通过kmem/smem.kmem/smem统计是用“fwctlpstat”.如果pstat很低的值不是fwkernel的问题.理解pstat是很重要的.-没有单个方面可以说明表明问题.,20,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,理解你的pstat,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Failuresin“fwctlpstat”,HMEM故障:只意味HMEM满了.配置问题HMEM应该更大.没有真正的内存问题.SMEM故障:可能的原因:达到了我们内存的极限.耗尽了OS的内存.大量non-sleep分配.Indicatesomeshortage.KMEM故障:应用需要内存但是没有得到.通常，这是一个内存问题.,22,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,额外的有趣的信息,23,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,资源利用率有没有一个例外！,23,24,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Resourcecapacitydoesnothavetobeasurprise!,24,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,包被丢弃,Itsfwsjobtodroppackets.Soitdoes.ButwhydoesitdroppacketsIlike?IsCPUusageveryhigh?如果任何CPU接近100忙碌，网卡将丢弃数据包。CheckwhytheCPUisbusy.OutOfMemory?OSCheckingforaresourcestarvedfirewall,showsnothing,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,多方面抓包,“fwmonitor”与其他sniffers有什么不同?在不同的位置得到Firewall-1内部的数据。可以准确的看到防火墙看到了什么.支持所有的平台Showpacketsaftervirtualdefragmentation.Mixedblessing.(喜忧参半)FilterpacketswithInspect.GettheconnectionUUID.使用FWmonitor去debugFirewall-1.什么模块丢弃了数据包?Firewall-1怎样改变了这个数据包?,27,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,fwmonitor抓包的位置,fwmonitor是可以在防火墙Chain中不同的位置抓包:,28,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,fwmonitorsyntax(fwmonitor语法),fwmonitor-e-m-oPacketsareinspectedinall4pointsunlessamaskisspecified(-moption).-d-turnondebugmode.-D-turnonverbosedebugmode.-e-specifyanINSPECTprogramline(multiple-eoptionscanbeused).-f-specifiesaINSPECTfilterfile.-l-specifyhowmuchofthepackettocapture(inbytes).-m-specifycapturepointsi,I,o,orOcanbeused-o-specifyanoutputfile.-x-specifydisplayparameteroffset.-pcapturepointinthechain(i.e.pall).,29,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,FWmonitor输出的一个例子,cpmodule#fwmonitormonitor:gettingfilter(fromcommandline)monitor:compilingmonitorfilter:CompiledOK.monitor:loadingmonitor:monitoring(control-Ctostop)eth0:i285:33-(TCP)len=285id=1075TCP:1050-18190.PA.seq=bf8bc98eack=941b05bceth0:I285:33-(TCP)len=285id=1075TCP:1050-18190.PA.seq=bf8bc98eack=941b05bceth0:o197:-33(TCP)len=197id=44599TCP:18190-1050.PA.seq=941b05bcack=bf8bca83eth0:O197:-33(TCP)len=197id=44599TCP:18190-1050.PA.seq=941b05bcack=bf8bca83eth0:o1500:-33(TCP)len=1500id=44600,30,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,阅读fwmonitor的输出,Line1:eth0:i285:33-(TCP)len=285id=1075第一个网卡(eth0)在入站方向beforethevirtualmachine(i)包的长度是285bytes包的IDis1075.packet从33发送到并且装载一个TCPheader/payload.,31,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Line2:TCP:1050-18190.PA.seq=bf8bc98eack=941b05bcTCPpayload在IPpacket内部从1050端口发送到18190.端口随后的部分显示了TCPflags设置(inthiscasePUSHandACK).最后的两个元素显示了TCPpacket的sequencenumber(seq=bf8bc98e)和这个acknowledgedsequencenumber(ack=941b05bc).,32,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,CPEthereal,MainscreenofCPEthereal,33,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,TCPStateEnforcement,第一个包必须为SYNRequiredforcorrectenforcementofstatefulfeaturesAconnectionthat“startsfromthemiddle“mightbe:injectedattackassociatedtowrongserviceerrorsinthenetwork但是,长时间idleconnections是有问题的。Solutions:IncreaseservicetimeoutAcceptoutofstateRejectoutofstateTCPpackets(fw_reject_non_syn),34,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Missinglogs?Timeouts?,35,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,DebuggingCorruptPackets,1)Definealargedebugbuffer:fwctldebug-buf320002)Turnondebugflagsthathelpbetterunderstandingthecontext:fwctldebug+vmconn3)Turnondroplogginginawaythatdumpsdroppedpacketsaswell:fwctlsetintfw_droplog_options0 x114)Startcollectingdebuginformationfromkernel,withtimestampenabled:fwctlkdebug-T-f5)Runfwmonitor-ocapt_filetocapturethecraftedpackets,36,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,KernelDebuggingTipsCyclicDebugFiles(循环调试文件),fwctlkdebug-f-o-m-snumberisthemaximumnumberoffilestocreatesizeisthemaximumsizeofeachfileOutputisbeprintedintofile_name.Whenitreachesthegivensize(moreorless),file_nameisrenamedtofile_name.0”andanewfile_nameiscreated.Iftheresalreadyfile_name.0itsrenamedtofile_name.1”,andsoon,untilthefilenumberlimitisreached.Then,theoldestfilesarejustdeleted.,37,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,DebuggingMechanismcont.,Youcancheckwhichdebugflagsareenabledbysimplyrunningfwctldebug不要忘记关闭debuggingfwctldebug0Thisonede-allocatesthebufferandautomaticallykills“fwctlkdebug”process,38,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,GlobalProperties?,39,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,TracingVPNForErrors,39,40,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,ServiceRelated?,41,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,NAT?,42,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,VoIP,42,43,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Troubleshootingcasestudy,Context(Customer:Telco)Provider-1UpgradeR55R65Gateways:VPN-1ClusterinR55replacedbynewPower-1appliancesinR65.AlltestsOKinAcceptancedocumentexceptforConnectiontable.,44,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Troubleshootingcasestudy-cont.,gw1#fwtab-tconnectionssHOSTNAMEID#VALS#PEAK#SLINKSlocalhostconnections81584060761217131198gw2#fwtab-tconnectionssHOSTNAMEID#VALS#PEAK#SLINKSlocalhostconnections8158217883945865268,45,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,gw1#fwctlpstatSyncVersion:newStatus:AbletoSend/ReceivesyncpacketsSyncpacketssent:total:10635435,retransmitted:0,retransreqs:0,acks:5Syncpacketsreceived:total:67912,werequeued:0,droppedbynet:0gw2#fwctlpstatVersion:newStatus:AbletoSend/ReceivesyncpacketsSyncpacketssent:total:266939,retransmitted:0,retransreqs:269,acks:181496Syncpacketsreceived:total:10862439,werequeued:257,droppedbynet:2,gw1#fwctlpstatSyncVersion:newStatus:AbletoSend/ReceivesyncpacketsSyncpacketssent:total:10635435,retransmitted:0,retransreqs:0,acks:5Syncpacketsreceived:total:67912,werequeued:0,droppedbynet:0gw2#fwctlpstatVersion:newStatus:AbletoSend/ReceivesyncpacketsSyncpacketssent:total:266939,retransmitted:0,retransreqs:269,acks:181496Syncpacketsreceived:total:10862439,werequeued:257,droppedbynet:2,gw1#fwctlpstatSyncVersion:newStatus:AbletoSend/ReceivesyncpacketsSyncpacketssent:total:10635435,retransmitted:0,retransreqs:0,acks:5Syncpacketsreceived:total:67912,werequeued:0,droppedbynet:0gw2#fwctlpstatVersion:newStatus:AbletoSend/ReceivesyncpacketsSyncpacketssent:total:266939,retransmitted:0,retransreqs:269,acks:181496Syncpacketsreceived:total:10862439,werequeued:257,droppedbynet:2,Troubleshootingcasestudy-cont.,46,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Troubleshootingcasestudy-cont.,Troubleshootingstepsinacceptancedocument:Note:在这点上没有kernelorservicedebug的debugAtthispointnokernelorservicedebug已经完成。Analysisofpacketcaptureonthebackupmodulehavebeendone(tcpdumpieth38116):OKAnalysisoffwtabuoutputtobedone.,47,UnrestrictedForeveryone,2009CheckPointSoftwareTechnologiesLtd.Allrightsreserved.,Troubleshootingcasestudy-cont.,gw1#Top10Services:=Service:53Hits:13430Rules:31,161,162,41,348,46,662Service:80Hits:1978Rules:1502,256,251,1132,592,1386,1215,1464,1678,Service:4433Hits:498Rules:897,1383,1105Service:7024Hits:290Rules:289,1698,1418,810Service:21Hits:190Rules:433,500,451,4