ppt课件-achieving high performance throughput in production networks

1,Achieving high performance throughput in production networks,Les Cottrell SLACPresented at the Internet 2 HENP Networking Working Group kickoff meeting at Internet 2 Ann Arbor, Michigan, Oct 26 01,Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP,/grp/scs/net/talk/thru-i2henp-oct01.html,2,High Speed Bulk Throughput,Driven by:Data intensive science, e.g. data gridsHENP data rates, e.g. BaBar 300TB/year, collection doubling yearly, i.e. PBytes in couple of years Data rate from experiment today 20MBytes/s 200GBytes/dMultiple regional computer centers (e.g. Lyon-FR, RAL-UK, INFN-IT, LBNL-CA, LLNL-CA, Caltech-CA) need copies of dataTier A gets 1/3 data in 1/3 year (full rate), SLAC does not keep copyBoeing 747 high throughput, BUT poor latency ( 2 weeks) & very people intensiveSo need high-speed networks and ability to utilizeHigh speed today = few hundred GBytes/day (100GB/d 10Mbits/s),Data vol,Moores law,3,How to measure network throughput,Selected about 2 dozen major collaborator sites in US, CA, JP, FR, CH, IT, UK over last yearOf interest to SLACCan get logon accountsUse iperf Choose window size and # parallel streamsRun for 10 seconds together with ping (loaded)Stop iperf, run ping (unloaded) for 10 secondsChange window or number of streams & repeatRecord # streams, window, throughput (Mbits/s), loaded & unloaded ping responses, cpu utilization, real timeVerify window sizes are set properly by using tcpdump cant believe what application tells you LMNote cpu speeds, interface speeds, operating system, path characteristics,4,Typical results,8kB,16kB,32kB,100kB,64kB,5,Windows vs Streams,Often for fixed streams*window product, streams are more effective than window size, e.g. SLACCERN, Jul 01:There is an optimum number of streams above which performance flattens outCommon for throughputs to be asymmetricmore congestion one way, different routes, host dependencies,6,Windows vs Streams,Multi-streams often more effective than windowsmore agile in face of congestionOften easier to set up Need root to configure kernel to set max windowNetwork components may not support big windowsSome OS treat max windows strangely DMay be able to take advantage of multiple pathsBut: may be considered over-aggressive (RFC 2914) pcan take more cpu cycleshow to know how many streams?,7,Iperf client CPU utilization,As expected increases with throughput (mainly kernel) d 0.7*MHz/Mbits/s,For fixed throughputFewer streams take less cpu J 6E.g. 1-4 streams take 20% less cpu than 8-16 streams for same throughput (if can get it),8,Throughput quality improvements,TCPBW IN2P3 and 340Mbps Caltech SLAC with GE,10,Network Simulator (ns-2),From UCB, simulates networkChoice of stack (Reno, Tahoe, Vegas, SACK)RTT, bandwidth, flows, windows, queue lengths Compare with measured resultsAgrees wellConfirms observations (e.g. linear growth in throughput for small window sizes as increase number of flows),11,Agreement of ns2 with observed,12,Ns-2 thruput & loss predict,Indicates on unloaded link can get 70% of available bandwidth without causing noticeable packet lossCan get over 80-90% of available bandwidthCan overdrive: no extra throughput BUT extra loss,90%,13,Simulator benefits,No traffic on network (nb throughput can use 90%)Can do what if experimentsNo need to install iperf servers or have accountsNo need to configure host to allow large windowsBUTNeed to estimate simulator parameters, e.g.RTT use ping or synackBandwidth, use pchar, pipechar etc., moderately accurateAND its not the real thingNeed to validate vs. observed dataNeed to simulate cross-traffic etc,14,Impact on Others,Make ping measurements with & without iperf loadingLoss loaded(unloaded)RTTLooking at how to avoid impact: e.g. QBSS/LBE, application pacing, control loop on stdev(RTT) reducing streams, want to avoid scheduling,15,File Transfer,Used bbcp (written by Andy Hanushevsky)similar methodology to iperf, except ran for file length rather than time, provides incremental throughput reports, supports /dev/zero, adding durationlooked at /afs/, /tmp/, /dev/nullchecked different file sizesBehavior with windows & streams similar to iperfThrubbcp 0.8*Thruiperf,For modest throughputs ( than for iperf,16,Application rate-limiting,Bbcp has transfer rate limitingCould use network information (e.g. from Web100 or independent pinging) to bbcp to reduce/increase its transfer rate, or change number of parallel streams,No rate limiting, 64KB window, 32 streams,15MB/s rate limiting, 64KB window, 32 streams,17,Using bbcp to make QBSS measurements,Run bbcp src data /dev/zero, dst=/dev/null, report throughput at 1 second intervalswith TOS=32 (QBSS) After 20 s. run bbcp with no TOS bits specified (BE)After 20 s. run bbcp with TOS=40 (priority)After 20 more secs turn off PriorityAfter 20 more secs turn off BE,18,QBSS test bed with Cisco 7200s,Set up QBSS testbedConfigure router interfaces3 traffic types:QBSS, BE, PriorityDefine policy, e.g.QBSS 1%, priority 30% Apply policy to router interface queues,10Mbps,100Mbps,100Mbps,100Mbps,1Gbps,Cisco 7200s,19,Example of effects,Also tried: 1 stream for all, and priority at 30%,20,QBSS with Cisco 6500,6500s + Policy Feature Card (PFC)Routing by PFC2, policing on switch interfaces2 queues, 2 thresholds eachQBSS assigned to own queue with 5% bandwidth guarantees QBSS gets somethingBE & Priority traffic in 2nd queue with 95% bandwidthApply ACL to switch port to police Priority traffic to factor of 10 larger RTT than unloadedIf both ping & iperf have QoS=Priority then ping RTT very variable since iperf limited to 30%RTT quick when iperf limited, long when iperf transmits,22,Possible HEP usage,Apply priority to lower volume interactive voice/video-conferencing and real time controlApply QBSS to high volume data replicationLeave the rest as Best EffortSince 40-65% of bytes to/from SLAC come from a single application, we have modified to enable setting of TOS bitsNeed to identify bottlenecks and implement QBSS thereBottlenecks tend to be at edges so hope to try with a few HEP sites,23,Acknowledgements for SC2001,Many people assisted in getting accounts, setting up servers, providing advice, software etc.Suresh Man Singh, Harvey Newman, Julian Bunn (Caltech), Andy Hanushevsky, Paola Grosso, Gary Buhrmaster, Connie Logg (SLAC), Olivier Martin (CERN), Loric Totay, Jerome Bernier (IN2P3), Dantong Yu (BNL), Robin Tasker, Paul Kummer (DL), John Gordon (RL), Brian Tierney, Bob Jacobsen, (LBL), Stanislav Shalunov (Internet 2), Joe Izen (UT Dallas), Linda Winkler, Bill Allcock (ANL), Ruth Pordes, Frank Nagy (FNAL), Emanuele Leonardi (INFN), Chip Watson (JLab), Yukio Karita (KEK), Tom Dunigan (ORNL), John Gordon (RL), Andrew Daviel (TRIUMF), Paul Avery, Greg Goddard (UFL), Paul Barford, Miron Livny (UWisc), Shane Canon (NERSC), Andy Germain (NASA), Andrew Daviel (TRIUMF), Richard baraniuk, Rold Reidi (Rice).,24,SC2001 demo,Send data from SLAC/FNAL booth computers (emulate a tier 0 or 1 HENP site) to over 20 other sites with good connections in about 6 countriesThroughputs from SLAC range from 3Mbps to 300MbpsPart of bandwidth challenge proposalSaturate 2Gbps connection to floor networkApply QBSS to some sites, priority to a few and rest Best EffortSee how QBSS works at high speedsCompeting bulk throughput streamsInteractive low throughput streams, look at RTT with ping,25,WAN thruput conclusions,High FTP performance across WAN links is possibleEven with 20-30Mbps bottleneck can do 100Gbytes/dayCan easily saturate a fast Ethernet interface over WANNeed GE NICs, OC3 WANs & to improve performancePerfo