HP解决方案中心性能调优方法论.ppt

何建波解决方案拓展部中国惠普有限公司Oct2004,PerformanceTuningMethodologyTechnology&Bestpractise,本文中的大部分内容是个人观点经验教训，仅供参考,2019/11/19,2,Concept,Whatisperformancetuning?ItisexpertsresponsibilitiesMeansworkingdayandnightMuchefforts,littlegains.性能优化是一个充满激情的工作，是一种爱的行为，真诚地帮助别人，.运用我们所知道的,所感觉到的,或者曾经承受过地来减轻他人的负担,2019/11/19,3,Concept,Whyperformancetuning/Testing?改善“客户体验”,提供一个可预期的系统降低I.T.成本,提供一个经济有效的系统NumberofCPUStoragecapacity(ZheJiangPublicsecurity-23%savings)不仅检验厂商的产品，而且考验people根本原因：Moderncomputer/applicationisnotfast/intelligenceenough,throughit,provideaintelligencesystem,2019/11/19,4,Concept,Whentuneperformance?新开发的系统&正在运行的生产系统Performancetuning?EnItisvendorsresponsibilitiesBeginsatsystemarchitectstage,followonoutlinedesign,coding,modificationandimplementationthroughallappslifecycle.IncludeApparchitect,EJBdesign,table/indexdesignBut,howtotunethesystemperformance?,2019/11/19,5,Methodology-How,0.与客户建立信任关系以询问开始Askquestionsfirst理解应用的体系结构（搭建与应用特性匹配的系统）使用记录工具确定“基准线”查看/检查系统隔离问题/简化问题每次改变一个参数（通常不可能）Knowwhentostoptuning,2019/11/19,6,1.Askquestionsfirst,磨刀不误砍材功，理解系统性能问题，了解症状明确目标了解历史，不知从哪里来，就不知道要到哪里去设计问题，询问技巧-找出在什么情况下发生了什么变化，产生了什么现象SMICfilenumberincreaseto3.5Millions,cpusysistoohighE-BUPTSMStesting,jvmversion,2019/11/19,7,系统架构（两层，多层？中间件？）逐层细化了解整个应用的体系架构，关键问题要追溯到函数调用一级业务/数据处理流程多少个(并发)用户（Keepconnectv.s.Active）系统/用户范围内的进程，线程数量、堆栈的需求虚拟和物理内存需求IPC(共享内存，信号灯，消息队列)打开的/文件Openedandmappedfilerequirements.HowmuchI/O?访问模式(顺序，随机？持续稳定的还是突发的)?Whatkindofnetworkingresourcesareneeded?(howmanysockets,),2.Understandapplicationarchitecture,2019/11/19,8,2.Understandapplicationarchitecture,flatfiles,process1,process2,process3,memory-residentdatabase,directory,signal,signal,directory,Oracleshadow,pipe,database,sharedmemory/semaphoresets,sigtimedwait(),sigtimedwait(),semop(),semop(),semop(),semop(),2019/11/19,9,3.使用监控工具,使用你适合/喜欢的工具记录系统的性能数据measureware/sar/glance/OVperformancemgt掌握系统的历史性能数据可以帮助你解决问题，优化系统进行对比正常情况/出现问题的对比优化步骤之间的对比易于roll-back进行容量规划通过检查系统的资源利用率和对应业务量的增减关系,2019/11/19,10,4.看什么？,第一印象系统资源利用率以及队列长度细心观察globalwaitstatesandglobalsystemcalls仔细研究processsystemcallsandprocesswaitsIstheprocessrunning?Ifnot,whatisitwaitingon?Ifitis,whereisitspendingittime?Whateveritisdoing,isitsupposetobedoingitthisway?,2019/11/19,11,5.确定问题域Whereistheproblem?,硬件？操作系统？数据库？中间件？应用？隔离问题上海浦东发展银行GuoXinLucent电信网管软件ICBC信贷台帐系统备份24小时哈尔冰商业银行轧张3小时简化问题,2019/11/19,12,6.对症下药cureit,方法直接定位法排除法一次改变一个地方观察结果时间安排Repeat,2019/11/19,13,6.何时结束,满足业务的需求，而不是系统资源利用率吞吐量响应时间系统中最昂贵的资源得以充分的利用CPUMEMI/O理解80/20原则,2019/11/19,14,Performancetuning-technology,FourlayerApplicationMiddlewareDatabaseO.S.H.W.ResourcesCPUMEMStorageNetwork,2019/11/19,15,Tools工欲善其事，必先利其器,2019/11/19,HPWorld2003SolutionsandTechnologyConference&Expo,16,ToolsforApplicationprofiling,Applicationlayer(tooidle,toobusy)太闲进程阻塞，系统资源利用率低（特别是CPU），哪里被阻塞？所需资源被谁占用？为何它不能及时释放？回答：whereisthatprocessspendingitstime?Whatkernelcodeisitexecuting?Whatuserspacecodeisitexecuting?Waitforwhat?WhohadtheresouceIneeded?ICBCcreditsystem,太忙资源利用率太高，进程的时间花在什么地方了？Whereisthethread/processspendingitstime?Kernelcodeactiveorsleeping?Userspaceactiveorsleeping?NeusoftInsuranceapplicationWhateveritsdoing,isitsupposetobedoingitthisway?Betweentheapplicationdevelopers,thecustomerandHP,somebodyhadbetterknow.我可以改用其它资源吗？ToolsTraceinapplicationsourcecode,notinH.W.,O.S.,Middleware,DBlevelForC,tusc,prospect,caliperForJava,HPJconfig,HPJmeter,HPJtuneDonotworkforaapplicationsbug!solutionsapplicationiteself,chatr,psrset,mpctl,wlm,prm,ICBCATM/SMICNeusoftInsuranceApplication-sorting,2019/11/19,HPWorld2003SolutionsandTechnologyConference&Expo,17,Prospect,Keytoperformanceis“howdoesmyapplicationspenditstime?”Prospectprovides:System-widesummary,yawn,butalsoPer-processandper-threaddata:Execution/blocked进程在哪里阻塞，阻塞多长时间Userspace(applicationandlibrary)andkernelprofileEveninstruction-levelprofiling!Providesper-CPUinformationUser,Idle,SystemInterrupt,TrapContextswitches.etc缺点：NouseforIObottlenecksIntimidatingoutput,2019/11/19,18,Caliper,GeneralpurposeperformanceanalysistoolforItaniumbasedsystemsWorksdirectlyonthebinaryas-isFeaturesVariousmetricsmeasurementEg:fprof,dtlb_miss,itlb_miss,dcache_miss,icache_miss,branch_prediction,func_coveretcPerformancedataandsourcecorrelationEasyinclusion/exclusionofspecificloadmodulesPerthread/aggregatedthreadreportMulti-processselectioncapabilityText/HTMLoutput,2019/11/19,19,Caliper-Metrics,VariousmetricsMeasureConfigurationFile-hotspotsfprof,cgprofcallgraphcgprofcache/TLBbehaviordcache_miss,icache_miss,dtlb_miss,itlb_missbranchpredictionbranch_predictioncallcountsfunc_count,cgproffunctioncoveragefunc_coverCPUeventstotal_cpucompilerfeedbackpbo,2019/11/19,20,Tusc,TraceUnixSystemCallsforaspecifiedprocessRuncommandundertusc,orattachtoPIDShowsSystemcalls,inputparametersandtheirreturnvaluesSystemcalldurationThreadidofthethreadmakingsystemcallsSignalinformation.etcUsage:tusc-T-f-e-l-oUsefuloptions:-cSummaryofsyscallcounts,errorsandCPUtime-cccCPUtimeforeverysyscall-T%TPrintatimestampbeforeeverytrace-rallDisplayreadbufferforallreads-wallDisplaywritebufferforallwrites-fFollowfork()s,2019年11月19日星期二,Backupslide,21,GlancePlusOptions(backupslide),GlancePlusCommandsMenuh-OnlineHelpq-exit(ore)A-ApplicationListg-ProcessListd-DiskReportP-PRMGroupLista-CPUByProcessori-IOByFileSystemY-GlobalSystemCallsc-CPUReportu-IOByDiskF-ProcessOpenFilesm-MemoryReportv-IOByLogicalVolumeM-ProcessMemoryRegionst-SystemTablesN-NFSGlobalActivityR-ProcessResourcesw-SwapSpacen-NFSBySystemW-ProcessWaitStatesB-GlobalWaitsl-NetworkByInterfaceL-ProcessSystemCallsZ-GlobalThreadsT-TransTrackery-ReniceProcessG-ProcessThreadsH-AlarmHistorys-SelectProcessI-ThreadResourceJ-ThreadWaitS-SelectDisk/NFS/Appl/Trans/Thread,2019/11/19,22,Performancetuning-Middleware,TransactionProcessmonitortuxedoMSSQ(seperatereplyandsendqueue)MSMQApplicationServerCoreserverWebApplicationsJDBCEJB,2019/11/19,23,CoreServer,JVMheapsize32bit/64bitgcThread-countBesttunedempiricallytuneforhighCPUutilizationandthroughputDonotoverdo!Effectmaybeconstrainedbypoolsizes(SLSB,MDB,connectionpool)Application/ModuleSpecificExecuteQueuesUsefulinpreventingdeadlocksUsefulinhandlingrequestburst/surges,2019/11/19,24,CoreServer,Chunktuning:Requiredonlyforlargerequests/responses可以相应的减少socketreads/writes的数量-Dweblogic.ChunkSizeand-Dweblogic.utils.io.chunkpoolsizeMainFactors:MTUandrequestsizeSetonbothclientandserverHighermemoryrequirementsNativeI/O:Javai/onotnecessarilyslowerthannativei/o,2019/11/19,25,WebContainer,Uncommonbutcripplingfactor:SynchronizationinapplicationcodeHTTPSessionpersistenceIfsmallpercentageofsessiondatachanges,isolatethatdataintoaseparateobject.,2019/11/19,26,WebContainer,DisableServletandJSPpagechecks:禁止jsp-parampageCheckSeconds=-1servlet-reload-check-secs=-1incontainer-descriptorServletReloadCheckSecs=-1inWebAppComponentelementinconfig.xmlForJSPs,bothofthesehavetobesetGainsof5%to8%havebeenobservedJSPcachingtagSupportbothoutputandinputcachingWLS-proprietary,2019/11/19,27,JDBC,JDBCDriver选择正确的JDBCDriver非常重要Oraclethin10gdriversignificantlyfasterthanthe9ithindriverConnectionPooling不要动态起数据库连接min-capacity=max-capacityConnectionPoolsize=同时进行数据库操作的线程数Bigoverhead:ConnectionLeakProfiling,CheckConnectionOnReserve/ReleasePrepared-statementcacheConnection-specificcacheofcompiledSQLstatementsRequiresuseofjava.sql.PreparedStatementDefaultis10,shouldbesettototalnumberofpreparedstatementsintheapplicationForEJBapplications,aroughcalculationisno.ofpreparedstatements(no.-of-finders+no.-of-field-groups+3)*(no.-of-CMPBeans)Actualnumbercouldbehigher,2019/11/19,28,EJB,GeneralUseprovensession-faadepatternAvoid“RequiresNew”transactionattributeUselocalinterfacesorcall-by-reference无状态会话Beans和消息驱动的BeansMax-beans-in-free-poolandInitial-beans-in-free-pool有状态会话BeansMax-beans-in-cachelargeenoughtohandleconcurrentusersAvoidSFSBreplication,2019/11/19,29,EJB,实体BeansCachingbiggestperformanceboosterCachingonlyhelpsforpk-findersandcachedreferencesChooseconcurrencystrategy,cachingandlockingbehaviorforthebestperformanceForexample,forREPEATABLE_READyoucoulduseDatabase-concurrency+use-select-for-update*Optimistic-concurrency+cache-between-transactions+read-verificationThesecondcombinationwouldbemoreperformant,becauseofcache-between-transactionsexclusiveDBlocksheldforaveryshorttimeduringcommitRead-onlybeans*Notsupportedforalldatabases,2019/11/19,30,EJB,CMPEntityBeansEnablebatchinsertsandupdatesenable-batch-operationsTurnoffinclude-updatesonfindersandturnoffexistence-checksTurnedonbydefaultturnoffifnotneededRelationship-cachingUsesmulti-leveljoinstoloadmultiplebeansusingasingleSQLstatementField-groupsAllowsgroupingcommonlyusedCMPfields,2019/11/19,31,Performancetuning-database,DatabasePrincipal仔细设计（redolog,tablespace,schema,table,index,storedprocedure）S.A.M.E./EVA/Oracle10gASM上（应用）下（I.T.基础设施）贯通、相应规划搭建一个适合应用特点的系统不要在麻袋片上绣花-底子要好,2019/11/19,HPWorld2003SolutionsandTechnologyConference&Expo,32,ToolsforDatabase/Oracle,STATSPACKESTAT/BSTATLiveintheRDBMSdirectorySnapshotofdatabaseresourceusageforaworkloadNotreal-timeOracleEnterpriseManagerTools(9i&10g)DiagnosticsPackTuningPackReal-timeTKPROFQueryAnalysis3rdpartyQuestsoftwareSpotlightSQL*labToad,2019/11/19,HPWorld2003SolutionsandTechnologyConference&Expo,33,Database/Oracleperformancetips,I/OintensiveApply放之四海而皆准的原则S.A.M.E.SGAsizing(过犹不及)db_cache_sizelog_bufferShared_poolPGAsizing并发用户数每个进程占用的内存大小8i,9i&10g排序区,2019/11/19,34,TodaysRAIDSets,SCSIBus1,SCSIBus2,SCSIBus3,SCSIBus4,SCSIBus5,SCSIBus6,RAID0Volume,RAID5Volume,RAID1Volume,从根本上消除瓶颈问题，提高吞吐量，读写IO工作量被平均地分布到每个diskgroup的所有磁盘上避免对应用程序和数据库复杂的分析调优过程增加磁盘，自动leveling,HPEVAS.A.M.E.的硬件实现,2019/11/19,35,Performancetuning-O.S.,O.S.principalDeconfigureunnecessarymoduleApache,NIS,NFS,security,Configurekernelparameterscorrectlyvm,proc,ipc,inet,fs,ioConfigureitasapplicationinstallationguide,2019/11/19,HPWorld2003SolutionsandTechnologyConference&Expo,36,ToolsforO.S.,O.S.toolsGlance(瑞士军刀)，top,sar,iostat,vmstatKprofile,ktrace,cyclemeter,kgmon,KItrace,spinwatch,ipcrm,ipcs,timex,ndd,rtprio,rtsched,tunefs/vxtunefs,fsadm,scsictl,intctl,gprof,fuser,lsof,kctuneStorageI/Oramdisk,diskbenchNetworkingnetperfnetstat,lanscan,lanshow,2019年11月19日星期二,Backupslide,37,GlancePlusOptions,GlancePlusCommandsMenuh-OnlineHelpq-exit(ore)A-ApplicationListg-ProcessListd-DiskReportP-PRMGroupLista-CPUByProcessori-IOByFileSystemY-GlobalSystemCallsc-CPUReportu-IOByDiskF-ProcessOpenFilesm-MemoryReportv-IOByLogicalVolumeM-ProcessMemoryRegionst-SystemTablesN-NFSGlobalActivityR-ProcessResourcesw-SwapSpacen-NFSBySystemW-ProcessWaitStatesB-GlobalWaitsl-NetworkByInterfaceL-ProcessSystemCallsZ-GlobalThreadsT-TransTrackery-ReniceProcessG-ProcessThreadsH-AlarmHistorys-SelectProcessI-ThreadResourceJ-ThreadWaitS-SelectDisk/NFS/Appl/Trans/Thread,2019/11/19,38,Performancetuning-H.W.,nPar,vPar,PRM/WLM,psrset,mpctlCantheH.W.satisfymyneeds?Howdouknow?DoestheappNUMAaware?DoesTPC-Cmatchallapplications?CPU#?Memorysizing?I/Oadaptor,Controller,DisksScalabilityissues?Scale-independsonapplicationspecScale-outMiddleware/RACProductspecificfeatures,SD/NUMA,EVA,XPNetworkI/O,2019/11/19,39,2019/11/19,40,Performancetuning-casestudy,CPUtipsCPU不忙，系统很慢济南铁路局opsBottleneck在其它地方CPUsys占用太多，系统很慢深圳华为,SMICCPU100%忙,系统很慢成都电力计费系统sybase中国工商总行信贷台帐系统Insurancemulti-tablejoinFTS-Noindex,2019/11/19,41,Performancetuning-casestudy,Memoryandswapswapmem_on-allowphysicalmemorysizetoexceedtheavailableswaps