Capacity-Management-John-Allspaw-容量规划.ppt

CapacityManagement,forWebOperations,JohnAllspawOperationsEngineering,thebookImwriting,?,RulesofThumbPlanning/ForecastingStupidCapacityTricks,(withsomeFlickrstatisticssprinkledin),bugs(disguisedascapacityproblems)edgecases(disguisedascapacityproblems)securityincidentsrealcapacityproblems*,*(shouldbethelastthingyouneedtoworryabout),Thingsthatcancausedowntime,Capacity!=Performance,ForgetaboutperformanceforrightnowMeasurewhatyouhaverightNOWDontcountonitgettinganybetter,ThankYouHPCIndustry!,AutomatedStuffScalableMetricCollection/Display,alotofgreatdeploymentandmanagementtrickscomefromthem,adoptedbywebops,GoodMeasurementTools,recordandstoremetricsin/outcustommetricseasilycomparelightweight-ish,Cloudsneedplanningtoo,MakesdeploymentandprocurementeasyandquickButcloudsarestillresourceswithcostsandlimits,justlikeyourownstuffBlack-boxes:youmayneedtopayevenmoreattentionthanbefore,Metrics,SystemStatistics,Metrics,“Application”Level,(photosprocessedperminute),(averageprocessingtimeperphoto),(apacherequests),(concurrentbusyapacheprocs),Metrics,App-levelmeetssystem-level,here,totalCPU=1.12*#busyapacheprocs(ymmv),2400,photosperminutebeinguploadedrightNOW(Tuesdayafternoon),Ceilings,themostamountof“work”yourresourceswillallowbeforedegradationorfailure,ForgetBenchmarking,Findyourceilings,whatyouhaveleft,Userealliveproductiondatatofindceilings,Production:“itslikealab,butbigger!”,Like:databaseceilings,replicationlag:bad!,Ceilings,35,000,photorequestspersecondonaTuesdaypeak,SafetyFactors,SafetyFactors,Ceiling*FactorofSafety=URLIMITZ,SafetyFactors,webserver!,“safe”ceiling85%CPU,SafetyFactors,85%totalCPU=76busyapacheprocs,SafetyFactors,YahooFrontPagelinktoChineseNewYearPhotos,(photorequests/second),(8%spike),Forecasting,Forecasting,FictionalExample:webservers,Forecasting,Fictionalexample:15webservers.1week.,peakoftheweek,.biggersample,6weeks.isolatethepeaks.,Forecasting,.”AddaTrendline”withsomedecentcorrelation.,Forecasting,now,Forecasting,15servers76busyapacheproclimit=1140totalprocs,Forecasting,(week#10,duh),(1140-726)/42.751=9.68,WritingexcelmacrosisboringAllwewantis“daysremaining”,soallweneedisthecurve-fit,ForecastingAutomation,Usetoautomatethecurve-fit,Forecasting,FictionalExample:storageconsumption,ForecastingAutomation,actualflickrstorageconsumptionfromearly2005,inGB(ceilingisfictional),ForecastingAutomation,cmdlinescript,output,jallspaw:$cfityk./fit-storage.fit1#Fitykscript.Fitykversion:0.8.220guessQuadraticNewfunction%_1wascreated.4fitInitialvalues:lambda=0.001WSSR=464.564#1:WSSR=0.90162lambda=0.0001d(WSSR)=-463.663(99.8059%)#2:WSSR=0.736787lambda=1e-05d(WSSR)=-0.164833(18.2818%)#3:WSSR=0.736763lambda=1e-06d(WSSR)=-2.45151e-05(0.00332729%)#4:WSSR=0.736763lambda=1e-07d(WSSR)=-3.84524e-11(5.21909e-09%)Fitconverged.Betterfitfound(WSSR=0.736763,was464.564,-99.8414%).5infoformulain0#storage-consumption14147.4+146.657*x+0.786854*x26quitbye.,ForecastingAutomation,(SAME),fitykgave:y=0.786854x2+146.657x+14147.4(R2=99.84)Excelgave:y=0.7675x2+146.96x+14147.3(R2=99.84),CapacityHealth,12,629nagioschecks1314hosts6datacenters4photo“farms”farm=2DCs(east/west),HighandLowWaterMarks,alertifhigher,alertiflower,Perserver,squidrequestspersecond,Agooddashboardlookssomethinglike.,(yes,fictionalnumbers),DiagonalScaling,ImageprocessingmachinesReplaceDellPE860swithHPDL140G3s,verticallyscalingyouralreadyhorizontalnodes,DiagonalScalingexample:imageprocessing,4cores,8cores,(aboutthesameCPU“usage”perbox),45images/minpeak,140images/minpeak,(sameCPUusage,but3xmorework)“processing”meansmaking4sizesfromoriginals,DiagonalScalingexample:imageprocessingthroughput,DiagonalScalingexample:imageprocessing,3.52,terabyteswillbeconsumedtoday(onaTuesday),2ndOrderEffects(bewarethewanderingbottleneck),runninghot,soaddmore,2ndOrderEffects(bewarethewanderingbottleneck),runninggreatnow,somoretraffic!,nowtheserunhot,StupidCapacityTricks,StupidCapacityTricksquickanddirtymanagement,DSH,rootnetmon101#catgroup.of.serverswww100www118dbcontacts3admin1admin2,StupidCapacityTricksquickanddirtymanagement,rootnetmon101#dsh-Ngroup.of.serversdshdateexecutingdatewww100:MonJun2314:14:53UTC2008www118:MonJun2314:14:53UTC2008dbcontacts3:MonJun2307:14:53PDT2008admin1:MonJun2314:14:53UTC2008admin2:MonJun2314:14:53UTC2008dsh,StupidCapacityTricksTurnStuffOFF,Disableheavy-ishfeaturesofthesite(on/offswitches)Wehave195differentthingstodisableincaseofemergency.,StupidCapacityTricksTurnStuffOFF,uploads(photo)uploads(video)uploadsbyemailvariousAPIthingsvariousmobilethingsvarioussearchthingsetc.,etc.,Hostyouroutage/status/blogpageinmorethanonedatacenter.Tellyouruse