NetFlowPeranceAnalysis.doc

NetFlow Performance AnalysisThe Cisco IOS NetFlow feature set allows for the tracking of individual IP flows as they are received at a Cisco router or switching device. Network administrators can use the NetFlow flow records for a variety of purposes, including accounting, billing, network planning, traffic engineering, and user or application monitoring.NetFlow services are available on Cisco IOS Software-based routers including Cisco 800 thru 7500Series Routers, as well as the Cisco Catalyst 6500 Series Switch, Cisco 7600, 10000, 12000Series Routers and CRS-1 devices.For more details on NetFlow, refer to the NetFlow Services Solutions Guide white paper at: /en/US/products/sw/netmgtsw/ps1964/products_implementation_design_guide09186a00800d6a11.htmlAlthough many Cisco customers want to deploy NetFlow services, they are naturally cautious aboutintroducing new technology into their network without completely understanding the potential performance impact. This paper examines the CPU impact of enabling NetFlow services in various scenarios on several different Cisco hardware platforms.This information is valuable for planning potential NetFlow implementations, but these results are not a replacement for proper customer lab testing, pilot deployments, and other types of solution validation.Testing MethodologyFor the following selection of routers, tests were performed using Cisco IOS Software Release 12.0S. The platforms and configurations tested include: Cisco 2600 Router Cisco 2851 Router Cisco 3640 Router Cisco 3745 Router Cisco 7200 Router with Network Processing Engine NPE-300 Cisco 7200 Router with Network Services Engine NSE-1 Cisco 7500 Router with Route Switch Processor 8 using Cisco Express Forwarding Cisco 7500 Router with Route Switch Processor 8 using Distributed Cisco Express Forwarding Cisco 12000 Internet Router running Distributed Cisco Express Forwarding (Engine 1) Cisco 12000 Internet Router running Distributed Cisco Express Forwarding, with 1:100 sampling enabledFor the following selection of routers an enhanced selection of tests were run using Cisco IOS Software Release 12.4T: Cisco 1841 Router Cisco 2811 Router Cisco 3845 Router Cisco 7200 Router with Network Processing Engine NPE-400 Cisco 7200 Router with Network Processing Engine NPE-G1 Cisco 7200 Router with Network Processing Engine NPE-G2 Cisco 7301 RouterSeventeen test cases were defined and not all are run on all the platforms listed. The test cases are documented below. A mnemonic has been assigned to each test case to make it easier to understand the charts and graphs in the Test Results and Test Analysis sections of this document. Table 1 describes the test cases.Table 1. NetFlow Test CasesMnemonicTest DescriptionBaselineBaseline test without NetFlow enabled; provides a context for the tests that followNF-loadStatistics generated immediately after NetFlow services are enabled on the router; tests any unusual initialization requirements of NetFlow. No traffic is running.NF-enableNetFlow version 5 enabled but no export destination defined; documents the effects of NetFlow on the router itselfNF-NDENetFlow version 5 enabled and NetFlow Data Export (NDE) destination also defined; tests the effects of NDE on the routerNF-NDE-2NetFlow version 5 enabled and two different NDE destinations also defined; tests the effects of NDE with multiple destinations on the routerv9-NDE1NetFlow version 9 enabled and NetFlow Data Export (NDE) destination also defined; tests the effects of NDE on the routerv9-NDE2NetFlow version 9 enabled and two different NDE destinations also defined; tests the effects of NDE with multiple destinations on the routerNF-NDE-ASNetFlow version 5 enabled and recording autonomous-system origin of packets; NDEdestination also defined; used to test the effects of maintaining information aboutautonomous systems with NetFlowV9-NDE-AS-NHNetFlow version 9 enabled and recording autonomous-system peer of packets and BGP next-hop; NDE destination also defined; used to test the effects of maintaining information about autonomous systems with NetFlowNF-Prefix-V8NetFlow configured with a version 8 prefix aggregation scheme (but no NDE); compare results with NetFlow version 5NF-AS-V8-NDENetFlow configured with a version 8 autonomous system aggregation scheme and NDE; compare results with NetFlow v5NF-AS-TOSNetFlow configured with a version 8 autonomous system- Type of Service (ToS) aggregation scheme and NDE; compare results with NetFlow version 5FNF-loadStatistics generated immediately after Flexible NetFlow services are enabled on the router; testsany unusual initialization requirements of Flexible NetFlowFNF-enableNetFlow-original on all interfaces, no export; documents the effects of NetFlow on the routeritselfFNF-NDE-ASNetFlow version 9 enabled and recording autonomous-system origin of packets; NDEdestination also defined; used to test the effects of maintaining information aboutautonomous systems with Flexible NetFlowFNF-NDE-AS-NHNetFlow version 9 enabled and recording autonomous-system peer of packets and BGP next-hop; NDE destination also defined; used to test the effects of maintaining information about autonomous systems with Flexible NetFlowEach test case was performed with three different IP flow sets: 10,000, 45,000, and 65,000 flows(this represents the number of unique IP flows that were seen by the router for each test). Forsome platforms 70,000 flows were tested instead of 65,000 flows. The flows were sent in a loop, so that the NetFlow cache was populated by the first iteration of the traffic stream and used for switching the packets on subsequent iterations.For the enhanced selection of tests, multicast traffic was always flowing and for certain platforms, an additional set of tests with 2,000 flows were also run. The enhanced selection of tests include Flexible NetFlow.To ensure accuracy of the results and to eliminate any anomalies, output of the tests was not collected until NetFlow had been running for ten minutes (the NF-load test case was the exception to this rule). Six samples of output were taken at one-minute intervals following that. The numbers presented in the Test Results portion of this document represent the average of those results.As the timeouts for each flow set are constant, the packet per second (pps) rates presented to the router are different for each flow set.Table 2. Flow Counts and PPSFlow CountPPS2,00027410,000139445,000632665,0008903Note that the testing provided a worst-case scenario in terms of the traffic flows seen by the routers, and the results must be viewed in that context.All packet sizes were 64 bytes, a scenario that tends to be more stressful on a router than a mixed traffic stream of various sizes.At each flow rate, there was no duplication of flows until the test began its second iteration through the testing loop.Test ResultsThis section presents the results of the test cases described in the previous section. Interpretations and conclusions to be drawn from the data are discussed in the Test Analysis and Conclusions section of this document.The results are presented in raw, tabular format, so that readers will have all available information, and can utilize the numbers to extrapolate the results into their own environment.CPU Load ResultsTables 2 through 18 give results from the CPU utilization tests.Table 3. CPU Utilization: Cisco 1841 SeriesMnemonic2,000 Flows10,000 Flows45,000 Flows65,000 FlowsBaseline461823NF-load2222NF-enable5103344NF-NDE5103445NF-NDE-25103445NF-NDE-AS5103445NF-Prefix-V85103445NF-AS-V8-NDE5103444NF-AS-TOS5113444V9-NDE-15103545V9-NDE-25103445V9-NDE-AS-NH5103545FNF-load2222FNF-enable5124358FNF-NDE-AS5124258FNF-NDE-AS-NH5124359Table 4. CPU Utilization: Cisco 2600 SeriesMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline161616NF-load254249NF-enable356268NF-NDE396469NF-NDE-2396470NF-NDE-AS396470NF-Prefix-V8406370NF-AS-V8-NDE406370NF-AS-TOS406570Table 5. CPU Utilization: Cisco 2811 SeriesMnemonic2,000 Flows10,000 Flows45,000 Flows65,000 FlowsBaseline472027NF-load2222NF-enable6123850NF-NDE6134253NF-NDE-26134153NF-NDE-AS6134153NF-Prefix-V86134153NF-AS-V8-NDE6134153NF-AS-TOS6134153V9-NDE-16134153V9-NDE-26134153V9-NDE-AS-NH6134153FNF-load2222FNF-enable7165269FNF-NDE-AS7155371FNF-NDE-AS-NH7155271Table 6. CPU Utilization: Cisco 2851 RouterMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline102225v5, NDE 1143648v5, NDE 2133649v9, NDE1153348v9, NDE2143760Table 7. CPU Utilization: Cisco 3640 RouterMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline637NF-load91315NF-enable163339NF-NDE173340NF-NDE-2173539NF-NDE-AS173339NF-Prefix-V8203541NF-AS-V8-NDE193341NF-AS-TOS173342Table 8. CPU Utilization: Cisco 3745 RouterMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline102328v5, NDE 1143547v5, NDE 2143551v9, NDE1143549v9, NDE2153760Table 9. CPU Utilization: Cisco 3845 RouterMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline234NF-load211NF-enable379NF-NDE479NF-NDE-2379NF-NDE-AS379NF-Prefix-V8379NF-AS-V8-NDE379NF-AS-TOS379V9-NDE-1379V9-NDE-2379V9-NDE-AS-NH379FNF-load111FNF-enable479FNF-NDE-AS479FNF-NDE-AS-NH479Table 10. Cisco 7200 NSE-1: CPU UtilizationMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline6812NF-load91926NF-enable91927NF-NDE91926NF-NDE-291927NF-NDE-AS91927NF-Prefix-V891927NF-AS-V8-NDE91926NF-AS-TOS91927Table 11. CPU Utilization: Cisco 7200 NPE 300Mnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline112735v5, NDE 1183538v5, NDE 2163543v9, NDE1173740v9, NDE2183851Table 12. CPU Utilization: Cisco 7200 with NPE-400Mnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline51115NF-load221NF-enable72026NF-NDE72027NF-NDE-272027NF-NDE-AS72027NF-Prefix-V872027NF-AS-V8-NDE72028NF-AS-TOS72027V9-NDE-172027V9-NDE-272027V9-NDE-AS-NH72027FNF-load222FNF-enable72128FNF-NDE-AS72129FNF-NDE-AS-NH72129Table 13. CPU Utilization: Cisco 7200 with NPE-G1Mnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline233NF-load111NF-enable379NF-NDE379NF-NDE-2379NF-NDE-AS379NF-Prefix-V8379NF-AS-V8-NDE379NF-AS-TOS379V9-NDE-1379V9-NDE-2379V9-NDE-AS-NH379FNF-load111FNF-enable368FNF-NDE-AS368FNF-NDE-AS-NH378Table 14. CPU Utilization: Cisco 7200 with NPE-G2Mnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline135NF-load100NF-enable258NF-NDE268NF-NDE-2268NF-NDE-AS268NF-Prefix-V8268NF-AS-V8-NDE268NF-AS-TOS268V9-NDE-1268V9-NDE-2268V9-NDE-AS-NH268FNF-load100FNF-enable268FNF-NDE-AS258FNF-NDE-AS-NH268Table 15. CPU Utilization: Cisco 7301Mnemonic2,000 Flows10,000 Flows45,000 Flows65,000 FlowsBaseline1233NF-load1111NF-enable1368NF-NDE1379NF-NDE-22379NF-NDE-AS2379NF-Prefix-V82379NF-AS-V8-NDE1378NF-AS-TOS2368V9-NDE-11379V9-NDE-21379V9-NDE-AS-NH2378FNF-load0111FNF-enable2378FNF-NDE-AS2378FNF-NDE-AS-NH2379Table 16. CPU Utilization: Cisco 7500, RSP8, Cisco Express ForwardingMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline235NF-load51521NF-enable51522NF-NDE51622NF-NDE-251622NF-NDE-AS51622NF-Prefix-V861523NF-AS-V8-NDE61522NF-AS-TOS61622Table 17. CPU Utilization: Cisco 7500, RSP8, Distributed Cisco Express ForwardingMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline6915NF-load92028NF-enable92028NF-NDE92028NF-NDE-292028NF-NDE-AS91928NF-Prefix-V892029NF-AS-V8-NDE92028NF-AS-TOS92028Note: Because Distributed Cisco Express Forwarding was employed in this test, the CPU utilization numbers were collected on the Versatile Interface Processors (VIPs), not the main CPU.Table 18. CPU Utilization: Cisco 12000, Distributed Cisco Express ForwardingMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline7811NF-load122330NF-enable122331NF-NDE122431NF-NDE-2122331NF-NDE-AS122431NF-Prefix-V8122532NF-AS-V8-NDE122331NF-AS-TOS122331Note: Because Distributed Cisco Express Forwarding was employed in this test, theCPUutilization numbers were collected on the line cards, not the main CPU.Table 19. CPU Utilization: Cisco 12000 Distributed Cisco Express Forwarding, 1:100 SamplingMnemonic10,000 Flows45,000 Flows65,000 FlowsBaseline7811NF-load91214NF-enable91214NF-NDE91214NF-NDE-291214NF-NDE-AS91214NF-Prefix-V891214NF-AS-V8-NDE91215NF-AS-TOS91214Note: As above, because Distributed Cisco Express Forwarding was employed in this test, the CPU utilization numbers were collected on the line cards, not the main CPU.NetFlow Export ResultsThe following tables provide information regarding the rate of NetFlow data export for selected test cases. It is important to note that in the tests, all traffic was initiated at the same time and therefore the timeouts for the flows would happen at the same time.Figure 1. Cisco 1841 NF-NDE 2k Flows Rate of Export (pps)Table 20. Cisco 1841, 2,000 Flows NetFlow Export RatesMnemonicExport Time (sec)pps (max/avg)Bps (max/avg)Records per secondNF-NDE1013 /520000 / 7600390 / 150NF-NDE-21015 / 523000 / 7600450 /150NF-NDE-AS911 / 617000 / 8400330 / 170V9-NDE-1917 / 625000 / 8900450 / 170V9-NDE-21014 / 620000 / 8200370 / 160V9-NDE-AS-NH1114 / 620000 / 8200350 / 140FNF-NDE-AS1218 / 624000 / 7400440 / 140FNF-NDE-AS-NH1020 / 727000 / 9700440 / 160Table 21. Cisco 3845, 65,000 Flows NetFlow Export RatesMnemonicExport Time (sec)pps (max/avg)Bps (max/avg)Records per secondNF-NDE6434 / 3451000 / 510001020 / 970NF-NDE-26434 / 3451000 / 510001020 / 1000NF-NDE-AS6434 / 3451000 / 510001020 / 1000V9-NDE-16437 / 3654000 / 520001050 / 1020V9-NDE-26637 / 3554000 / 510001050 / 990V9-NDE-AS-NH6539 / 3857000 / 490001050 / 1000FNF-NDE-AS6740 / 3956000 / 540001000 / 970FNF-NDE-AS-NH6744 / 4262000 / 590001012 / 1000Figure 2. Cisco 7200 NPE-G1 NF-NDE, 65,000 Flows NetFlow Rate of ExportTable 22. Cisco 7200 NPE-G1, 65,000 Flows NetFlow Export RatesMnemonicExport Time (sec)Pps (max/avg)Bps (max/avg)Records per secondNF-NDE6434 / 3451000 / 510001020 / 1015NF-NDE-26634 / 3351000 / 500001020 / 985NF-NDE-AS6734 / 3251000 / 490001020 / 970V9-NDE-16537 / 3554000 / 510001050 / 1004V9-NDE-26637 / 3554000 / 510001049 / 998V9-NDE-AS-NH6739 / 3757000 / 540001046 / 974FNF-NDE-AS6740 / 3956000 / 540001000 / 970FNF-NDE-AS-NH6744 / 4262000 / 590001012 / 970Test Analysis and ConclusionsNetFlow CPU Utilization versus BaselineAs mentioned in the introduction, customers need to understand the potential performance impact of enabling NetFlow before they are willing to deploy it. The first series of charts in this section examines test cases and illustrates the effect of NetFlow on CPU utilization.In terms of additional CPU utilization (over and above the baseline), a few trends can easily be discerned. As the number of flows increases, the delta between the baseline and NetFlow-enabled CPU utilization widens. In other words, the more IP flows present, the more system resources NetFlow requires. Although these results were expected, they confirm the accuracy of the expectation. The more active flows NetFlow is maintaining in its cache, the larger the cache becomes and the more CPU it requires to sort through the cache.Note that in the figures below, CPU utilization does not seem to vary greatly, depending on the particular NetFlow features that are enabled. Neither the recording of autonomous-system numbers nor the addition of NetFlow data export (even to multiple destinations) makes a large impact on overall CPU utilization. Not surprisingly, the Cisco 2600 Router seems to vary the most in this regard.Figure 3. Cisco 1841 RouterFigure 4. Cisco 2600 RouterFigure 5. Cisco 2811 RouterFigure 6. Cisco 2851 RouterFigure 7. Cisco 3640Figure 8. Cisco 3745Figure 9. Cisco 3845Figure 10. Cisco 7200 NPE 400: CPU UtilizationFigure 11. Cisco 7200 NSE-1: CPU UtilizationFigure 12. Cisco 7200 NPE-300: CPU UtilizationFigure 13. Cisco 7200 NPE-400: CPU UtilizationFigure 14. Cisco 7301 CPU UtilizationFigure 15. Cisco 7500 RSP8 Cisco Express Forwarding: CPU UtilizationFigure 16. Cisco 7500 RSP 8 Distributed Cisco Express Forwarding: CPU UtilizationFigure 17. Cisco 12000 Distributed Cisco Express Forwarding: CPU Utili