CUDA Optimization NVIDIA培训资料

CUDAOptimization PengWang Ph D HPCDeveloperTechnology NVIDIA OptimizationOverview GPUarchitectureKerneloptimizationLatencyoptimizationMemoryoptimizationInstructionoptimizationDatatransferoptimizationOverlappedexecutionusingstreams GPUArchitecture GPUHighLevelView SMEM SMEM SMEM SMEM GlobalMemory CPUChipset PCIe StreamingMultiprocessorGlobalmemoryx bar FermiMultiprocessor Controlunit2WarpSchedulerUpto1536concurrentthreadsExecutionunit32CUDACoresFullIEEE754 2008FP32andFP6432FP32ops clock 16FP64ops clockSFU LSUMemoryRegisters 32bit 32K32 bitCacheL1 sharedmemory 64KB Texture 8KB Constant 8KB GPUandProgrammingModel Software GPU Threadsareexecutedbyscalarprocessors Thread CUDACore ThreadBlock Multiprocessor ThreadblocksareexecutedonmultiprocessorsThreadblocksdonotmigrateSeveralconcurrentthreadblockscanresideononemultiprocessor limitedbymultiprocessorresources Grid Device AkernelislaunchedasagridofthreadblocksUpto16kernelscanexecuteonadeviceatonetime WarpandSIMT Block 32Threads 32Threads 32Threads Warps Blocksdivideintogroupsof32threadscalledwarpsWarpsarebasicschedulingunitsWarpsalwaysperformthesameinstruction SingleInstructionMultipleThreads SIMT AlotofwarpscanhidememorylatencyContextswitchingisfree Time FermiMemoryHierarchy 3levels verysimilartoCPURegisterSpillstolocalmemoryCachesSharedmemoryL1cacheL2cacheConstantcacheTexturecacheGlobalmemory FermiMemoryHierarchyReview L2 GlobalMemory Registers C SM 0 L1 SMEM TEX Registers C SM 1 L1 SMEM TEX Registers C SM N L1 SMEM TEX GeneralOptimizationStrategies Measurement FindoutthelimitingfactorinkernelperformanceMemorybandwidthbound memoryoptimization Instructionthroughputbound instructionoptimization Latencybound configurationoptimization Measureeffectivememory instructionthroughputOptimizeforpeakmemory instructionthroughputFindingoutthebottleneckTypicallyaniterativeprocess KernelOptimizationWorkflow FindLimiter ComparetopeakGB s Memoryoptimization Comparetopeakinst s Instructionoptimization Latencyoptimization Memorybound Instructionbound Latencybound Done LatencyOptimization LatencyOptimization WhenthecodeislatencyboundBoththememoryandinstructionthroughputsarefarfromthepeakLatencyhiding switchingthreadsPurpose haveenoughthreadstohidelatencyMajortechniques adjustresourceusagetoincreasethreads UnderstandingLatencyBound Hardware inst mem ispipelinedLatency length ofthepipelineThroughput width ofthepipelineLittle sLawConcurrency Throughput LatencyBasicunits instruction cacheline thatareactiveinthepipeline on the fly C2050 150GB s 600 1 15GHz 128B 14SM 44cachelineperSMonthefly input output Latency 4Throughput 2Concurrency 8 Example Little sLawonGPU AchievedbandwidthasafunctionofrequestsperSM EnoughBlocks ofblocks ofSM 100toscalewelltofuturedeviceBlocksizeshouldbeamultipleof32 warpsize Minimum 64 Igenerallyuse128or256 Butusewhateverisbestforyourapp Dependsontheproblem doexperiments EnoughThreads GPUthreadsareverycheap useasmanyasyouwantOccupancy ratioofactivewarpsperSMtothemaximumnumberofallowedwarpsMaximumnumber 48inFermiWeneedtheoccupancytobehighenoughtohidelatency WhatLimitsOccupancy PartitioningofSMResourcesSharedmemoryispartitionedamongblocksRegistersarepartitionedamongthreads 63Threadblockslots 8Threadslots 1536AnyofthosecanbethelimitingfactoronhowmanythreadscanbelaunchedatthesametimeonaSM OccupancyOptimizations KnowthecurrentoccupancyVisualprofiler ptxas options v outputresourceusageinfo inputtoOccupancyCalculatorAdjustresourceusagetochangeoccupancy AdjustblocksizeLimitregisterusageCompileroption maxrregcount n perfile launch bounds perkernelUsetemplateReducesharedmemoryDynamicalallocation OccupancyCalculator LatencyHidingOccupancyCalculation Assumeglobalmemorytakes400cycles weneed400 2 200arithmeticinstructionstohidethelatency Forexample assumethecodehas8independentarithmeticinstructionsforeveryoneglobalmemoryaccess Thus200 8 26warpswouldbeenough 54 occupancy Lessons RequiredoccupancydependsonBOTHarchitectureandapplicationInthisexample beyond54 higheroccupancywon tleadtofurtherperformanceincrease MemoryOptimization MemoryOptimization Ifthecodeismemory boundandeffectivememorythroughputismuchlowerthanthepeakPurpose accessonlydatathatareabsolutelynecessaryMajortechniquesImproveaccesspatterntoreducewastedtransactions coalescingReduceredundantaccess sharedmemory GlobalMemoryThroughputMetric Measuringeffectivememorythroughput Fromtheapppointofview useful bytes numberofbytesneededbythealgorithmdividedbykerneltimeComparetothetheoreticalbandwidth70 80 isverygoodFindingoutbottleneckStartwithglobalmemoryoperations achievegoodthroughputAddarithmetic sharedmemory etc measuringperfasyougo Coalescing Globalmemorylatency 400 800cycles Thesinglemostimportantperformanceconsideration Coalescing globalmemoryaccessfromawarpcanbecoalescedintoasingletransactionCriterion requestsfromawarpfallinginaL1cacheline onetransaction transaction L1lineaccessed Load Warprequests32aligned consecutive4 bytewordsAddressesfallwithin1cache lineWarpneeds128bytes128bytesmoveacrossthebusonamiss addressesfromawarp 96 192 128 160 224 288 256 32 64 352 320 384 448 416 Memoryaddresses 0 Load 96 192 128 160 224 288 256 32 64 352 320 384 448 416 Memoryaddresses addressesfromawarp 0 Warprequests32aligned permuted4 bytewordsAddressesfallwithin1cache lineWarpneeds128bytes128bytesmoveacrossthebusonamiss Load 96 192 128 160 224 288 256 addressesfromawarp 32 64 0 352 320 384 448 416 Memoryaddresses Warprequests32misaligned consecutive4 bytewordsAddressesfallwithin2cache linesWarpneeds128bytes256bytesmoveacrossthebusonmisses Load addressesfromawarp 96 192 128 160 224 288 256 32 64 352 320 384 448 416 Memoryaddresses 0 Allthreadsinawarprequestthesame4 bytewordAddressesfallwithinasinglecache lineWarpneeds4bytes128bytesmoveacrossthebusonamiss Load addressesfromawarp 96 192 128 160 224 288 256 32 64 352 320 384 448 416 Memoryaddresses 0 Warprequests32scattered4 bytewordsAddressesfallwithinNcache linesWarpneeds128bytesN 128bytesmoveacrossthebusonamiss SharedMemory Lowlatency afewcyclesHighthroughput 73 6GB sperSM 1 03TB sperGPU Usage shared MainuseSharingdataamongthreadsofthesameblockUser managedcache SharedMemoryExample MatrixMultiplication A B C C AxB EverythreadcorrespondstooneentryinC NaiveKernel global voidsimpleMultiply float a float b float c intN introw threadIdx x blockIdx x blockDim x intcol threadIdx y blockIdx y blockDim y floatsum 0 0f for inti 0 i N i sum a row N i b i N col c row N col sum EverythreadcorrespondstooneentryinC BlockedMatrixMultiplication A B C C AxB Datareuseintheblockedversion Blockedandcachedkernel global voidcoalescedMultiply double a double b double c intN shared floataTile TILE DIM TILE DIM shared doublebTile TILE DIM TILE DIM introw blockIdx y blockDim y threadIdx y intcol blockIdx x blockDim x threadIdx x floatsum 0 0f for intk 0 k N k TILE DIM aTile threadIdx y threadIdx x a row TILE DIM threadIdx x bTile threadIdx y threadIdx x b threadIdx y N col syncthreads for inti k i k TILE DIM i sum aTile threadIdx y i bTile i threadIdx x c row N col sum PerformanceResults M N K 512 MemoryOptimizations StriveforperfectcoalescingTransposethedatastructure e g AOStoSOAPaddingChangeparallelizationscheme 1 thread per taskto1 warp per task Usesharedmemorytoreduceglobalmemoryaccess avoidnon coalescedaccessBoundtotexturecacheforunpredictableuncoalescedaccessUseconstantcacheifallthreadsinawarpwillaccessthesameconstantdata InstructionOptimization InstructionOptimization IfyoufindoutthecodeisinstructionboundCompute intensivealgorithmcaneasilybecomememory boundifnotcarefulenoughTypically worryaboutinstructionoptimizationaftermemoryandexecutionconfigurationoptimizationsPurpose reduceinstructioncountUselessinstructionstogetthesamejobdoneMajortechniquesReducewastedinstructions branchdivergence etc Usehighthroughputinstructions ControlFlow Singleinstructionmultiplethreads SIMT modelAsingleinstructionisissuedforawarpatatimeDifferentcodepathwithinawarphandledbyhardwareDivergentbranches threadswithinasinglewarptakedifferentpathsExamplewithdivergence if threadIdx x 2 else DifferentexecutionpathswithinawarpareserializedDifferentwarpscanexecutedifferentcodewithnoimpactonperformance Example if else NoBranchDivergencewithinaWarp Time Warp Warp BranchDivergencewithinaWarp Time UsingHighThroughputInstructions AvoidautomaticconversionofdoubletofloatAdding f tofloatingliterals e g 1 0f becausethedefaultisdoubleFermidefault ftz false prec div true prec sqrt trueforIEEEcomplianceFastmathfunctionsTwotypesofruntimemathlibraryfunctionsfunc slowerbuthigheraccuracy 5ulporless func fastbutloweraccuracy seeprog guideforfulldetails use fast math forceseveryfunc to func IntdivideandmoduloareexpensiveDivideby2 n use n Modulo2 n use 2 n 1 KernelOptimizationWorkflow FindLimiter ComparetopeakGB s Memoryoptimization Comparetopeakinst s Instructionoptimization Latencyoptimization Memorybound Instructionbound Latencybound Done DataTransferOptimization MinimizingCPU GPUdatatransfer Hostdevicedatatransferhasmuchlowerbandwidththanglobalmemoryaccess 8GB s PCIex16Gen2 vs156GB s 515Ginst s C2050 MinimizetransferIntermediatedatacanbeallocated operated de allocateddirectlyonGPUSometimesit sevenbettertorecomputeonGPUMoveCPUcodestoGPUthatdonothaveperformancegainsifitcanreducedatatransferGrouptransferOnelargetransfermuchbetterthanmanysmallones 10microseclatency 8GB s latencydominatedifdatasize 80KBOverlapmemorytransferwithcomputationDoublebuffering Overlapkernelandmemorycopy Requirements D2HorH2Dmem