V4ACITDM-G-RoCE

ACIRoCEv2Version1.1,MatthiasWessendorf,OverviewImplemtationinACIConfiguration,Agenda,RemoteDirectMemoryAccessProtocol(RDMA)AcceleratedIOdeliverymodelwhichworksbyallowingapplicationsoftwaretobypassmostlayersofsoftwareandcommunicatedirectlywiththehardwareRDMAbenefitsLowlatencyHighthroughputZerocopycapabilityOS/StackbypassRDMAHardwareTechnologiesInfinibandiWARP:RDMAoverTCP/IPRoCE:RDMAoverConvergedEthernet,WhatisRDMA?,FileServer,SMBDirect,Client,RDMANIC,SMBDirect,EthernetorInfiniBand,SMBServer,SMBClient,Memory,Memory,NDKPI,NDKPI,RDMANIC,RDMA,RoCE:RDMAoverConvergedEthernetProvideInfiniBand-likeperformanceandefficiencytoubiquitousEthernetinfrastructure.UtilizethesametransportandnetworklayersfromIBInfiniBandstackandswapthelinklayerforEthernet.ImplementIBverbsoverEthernet.(L2),WhatisRoCE,RoCEFrameFormat,RoCEStack,ComparingRDMATechnologies,*ThisiscurrentasofthereleaseofWindowsServer2012RC.Informationonthisslideissubjecttochangeastechnologiesevolveandnewcardsbecomeavailable.,ChangeswithRoCEv2,RoCEv2akaRouteableRoCE,RoCEv2Stack,ExampleNetworkTopologies(SMBDirect),OverviewImplementationinACIConfiguration,Agenda,DesignedtosatisfyrequirementsofSANandHPCCapplications.Lowlatency,Losslessswitching.FollowingenhancementsaddedtoEthernettomakeitsuitableforSANandHPCCtraffic.802.1QbbPriority-basedFlowControl802.1QauCongestionNotification802.1QazEnhancedTransmissionSelection802.1QazDCBExchangeProtocol,Standards:ConvergedEnhancedEthernet,ACIsupportsROCEv2byCongestionavoidanceandmanagementusingWREDwithECN:No-droptrafficclassusingPFC,ACIROCEv2design,CongestionManagementissupportedonTorsandSpines.CongestionManagementAlgorithmstosupportedTailDrop,WRED.DefaultCongestionAlgorithmwillbetaildrop.WREDWeightedRandomEarlyDropwilldrop/markpacketswithauser-specifiedprobabilityonceegressqueuescrossuser-specifiedqueuethreshold.NeededforECN,CongestionManagement,CongestionNotification,IPExplicitCongestionNotification(ECN)isusedforcongestionnotification.ECNenablesend-to-endcongestionnotificationbetweentwoendpointsonaIPnetworkIncaseofcongestion,ECNgetstransmittingdevicetoreducetransmissionrateuntilcongestionclears,withoutpausingtraffic.InROCEv2,thesendersendstheECNcapabletrafficbysettingIPTOSbitsto10or01.Whenthequeueiscongested,ACIswitchessetstheCongestionExperienced(CE)bitinIPTOSbitsas11.,PFCcanbeusedtoachieveLink-LayerFlowControl.PriorityFlowControlprovidesthecapabilitytopausetrafficforapre-determinedqueueincaseofcongestiononthelink.Priortothisfeature,PFCwassupportedonlyonTORs.ThisfeaturesextendsPFCsupporttospinesaswell.DCBXcanbeusedtocommunicateCOSprioritiesandotherrequiredinformationtotheROCE-enabledNIC.,PriorityFlowControl,NeedtoaddPFCsupportonallportsofLC/FCandsingle-chipspines.WhenPFCisenabledforatleastoneclass,allfabricandinternalportswillbecomePFCenabled.Dcbxsupportnotrequiredonspine.Needtosupport2no-dropclasses.CablelengthbetweenSpineandTor100metres.,FlowControlSupportonSpines,PFCisalreadysupportedonTorsonFront-PanelPorts.ThisfeaturesaddssupportforFabricPorts.WhenPFCisenabledforatleastoneclass,allfabricportswillbecomePFCenabled.DcbxsupportalreadypresentonFront-Panelports,norequirementtoaddsupportonFabricPorts.Sofarnumberofno-dropclassescurrentlysupportedis1.Thisfeatureincreaseditto2classes.DefaultcablelengthbetweenTorandServer100metres.DefaultcablelengthbetweenTorandSpine100metres.,FlowControlSupportonTOR,AminimumbandwidthguaranteeisrequiredbyROCEv2.Thisensuresthatno-droptrafficisnotcontinuallypausedduetoothertrafficclassesutilizingallthebandwidthonalink.ACIhasbandwidthpercentageconfigurableonGUIforLevel1-7traffic.WhenPFCisenabledononeofthese,bandwidthpercentagecanbeconfiguredtoguaranteeminimumbandwidth.,NetworkDesignConsiderations-QOS,Notsupportedfor1stGenPlatformsonlyEXandLaterCurrentlyonlyvalidatedonmodularspinewithEXlinecardsECNisnotsupportedonFEX.Onlyfullmeshtopologysupportedinthisrelease.M-topoandstretch-topotobesupportedinfuturereleases.ROCEv2isnotsupportedonMultipod/multisiteingangarelease.,Guidelines,OverviewImplementationinACIConfiguration,Agenda,ClassLevelConfigurationOptionsPFCEnable/DisableNo-DropcosvalueEnablescopefabric/front-panelportsofTor(NewlyAdded).PortLevelConfigurationOptions(TorFront-PanelOnly)PFCEnable/Disable,FlowControlConfigurationOptions,Flowcontrol/PFCConfigurationonToR,ConfigurationviaAPICUIclasslevelWRED,ECNandPFCforFabric,DefaultisToR,ConfigurationviaAPICCLI:WRED,ECN,apic1#configapic1(config)#qosparameterslevel2apic1(config-qos)#algowredapic1(config-qos-algo)#ecnenabledapic1(config-qos-algo)#maxthreshold80apic1(config-qos-algo)#minthreshold20apic1(config-qos-algo)#probability20apic1(config-qos-algo)#weight1,ConfigurationviaAPICCLI:PFC,apic1(config-qos)#pause?no-dropConfigureclassbasedflowc