性能测试之循序渐进.docx

附录 A:案例学习性能监控之循序渐进某一天,一个客户打电话来需要技术帮助,并抱怨平常 15 秒就可以打开的网页现在需要 20 分钟才可以打开.具体系统配置如下:RedHatEnterpriseLinux3update7Dell1850DualCoreXenonProcessors,2GBRAM,75GB15KDrivesCustomLAMPsoftwarestack(Llinux+apache+mysql+php环境)性能分析之步骤1.首先使用vmstat查看大致的系统性能情况:#vmstat110procsmemoryswapiosystemcpurb swpd free buff cache si so bi bo in cs us sy id wa10249844191441853212212120 0 7 3 22 17 25 8 17 1801249844178281852812226960 0 40448 8 1384 113813 7 65 1401249844180041852812227560 0 13568 4 623 534 3 4 56 3720249844178401852812232000 0 35200 0 1285 101717 7 56 2010249844224881852812186080 0 38656 0 1294 103417 7 58 180124984421228185441219908001369648460955953543801249844177521854412233760036224414691035106671711249844178561854412085200028724095094133124971024984417748185441222468004096881266116417959161024984417912185441222572004134412123710801386513分析:1,不会是内存不足导致,因为 swapping始终没变化(si和 so).尽管空闲内存不多(free),但swpd也没有变化.2,CPU方面也没有太大问题,尽管有一些运行队列(procsr),但处理器还始终有 50%多的 idle(CPUid).3,有太多的上下文切换(cs)以及 diskblock 从 RAM 中被读入(bo).4,CPU还有平均 20%的 I/O等待情况.结论:从以上总结出,这是一个 I/O瓶颈.2.然后使用iostat检查是谁在发出 IO请求:#iostatx1Linux2.4.2140.ELsmp()03/26/2007avgcpu:%user%nice%sys%idle30.000.009.3360.67Device:rrqm/swrqm/sr/sw/srsec/swsec/srkB/swkB/savgrqszavgquszawaitsvctm%util/dev/sda7929.0130.341180.9114.237929.01357.843964.50178.926.930.390.030.066.69/dev/sda12.675.460.401.7624.6257.7712.3128.8838.110.062.781.770.38/dev/sda20.000.300.070.020.572.570.291.2832.860.003.812.640.03/dev/sda37929.0124.581180.4412.457929.01297.503964.50148.756.900.320.030.066.68avgcpu:%user%nice%sys%idle9.500.0010.6879.82Device:rrqm/swrqm/sr/sw/srsec/swsec/srkB/swkB/savgrqszavgquszawaitsvctm%util/dev/sda0.000.001195.240.000.000.000.000.000.0043.693.600.99117.86/dev/sda10.000.000.000.000.000.000.000.000.000.000.000.000.00/dev/sda20.000.000.000.000.000.000.000.000.000.000.000.000.00/dev/sda30.000.001195.240.000.000.000.000.000.0043.693.600.99117.86avgcpu:%user%nice%sys%idle9.230.0010.5579.22Device:rrqm/swrqm/sr/sw/srsec/swsec/srkB/swkB/savgrqszavgquszawaitsvctm%util/dev/sda0.000.001200.370.000.000.000.000.000.0041.652.120.99112.51/dev/sda10.000.000.000.000.000.000.000.000.000.000.000.000.00/dev/sda20.000.000.000.000.000.000.000.000.000.000.000.000.00/dev/sda30.000.001200.370.000.000.000.000.000.0041.652.120.99112.51分析:1,看上去只有/dev/sda3分区很活跃,其他分区都很空闲.2,差不多有 1200读 IOPS,磁盘本身是支持 200IOPS 左右(参考之前的 IOPS计算公式).3,有超过 2 秒,实际上没有一个读磁盘(rkb/s).这和在vmstat看到有大量 I/Owait 是有关系的.4,大量的 readIOPS(r/s)和在vmstat中大量的上下文是匹配的.这说明很多读操作都是失败的.结论:从以上总结出,部分应用程序带来的读请求,已经超出了 I/O子系统可处理的范围.3.使用 top来查找系统最活跃的应用程序#topd111:46:11up3days,19:13,1user,loadaverage:1.72,1.87,1.80176processes:174sleeping,2running,0zombie,0stoppedCPUstates:cpuusernicesystemirqsoftirqiowaitidletotal12.8%0.0%4.6%0.2%0.2%18.7%63.2%cpu0023.3%0.0%7.7%0.0%0.0%36.8%32.0%cpu0128.4%0.0%10.7%0.0%0.0%38.2%22.5%cpu020.0%0.0%0.0%0.9%0.9%0.0%98.0%cpu030.0%0.0%0.0%0.0%0.0%0.0%100.0%Mem:2055244kav,2032692kused,22552kfree,0kshrd,18256kbuff1216212kactv,513216kin_d,25520kin_cSwap:4192956kav,249844kused,3943112kfree1218304kcachedPIDUSERPRNIVIRTRESSHRS%CPU%MEMTIME+COMMAND14939mysql250379M224M1117R38.225.7%15:17.78mysqld4023root1502120972784R2.00.30:00.06top1root1502008688592S0.00.20:01.30init2root3419000S0.00.00:22.59ksoftirqd/03rootRT0000S0.00.00:00.00watchdog/04root105000S0.00.00:00.05events/0分析:1,占用资源最多的好像就是mysql进程,其他都处于完全 idle状态.2,在 top(wa)看到的数值,和在vmstat看到的wio数值是有关联的.结论:从以上总结出,似乎就只有mysql进程在请求资源,因此可以推论它就是导致问题的关键.4.现在已经确定是mysql在发出读请求,使用strace来检查它在读请求什么.#stracep14939Process14939attachedinterrupttoquitread(29,3123713663371222%4200000012P/d,20)=20read(29,ata1/strongmail/log/strongmaild.,399)=399_llseek(29,2877621036,2877621036,SEEK_SET)=0read(29,112413663371223%4200000012P/da,20)=20read(29,ta1/strongmail/log/strongmailde.,400)=400_llseek(29,2877621456,2877621456,SEEK_SET)=0read(29,112353663371224%4200000012P/da,20)=20read(29,ta1/strongmail/log/strongmailde.,396)=396_llseek(29,2877621872,2877621872,SEEK_SET)=0read(29,112453663371225%4200000012P/da,20)=20read(29,ta1/strongmail/log/strongmailde.,404)=404_llseek(29,2877622296,2877622296,SEEK_SET)=0read(29,3123623663371226%4200000012P/d,20)=20分析:1,大量的读操作都在不断寻道中,说明mysql进程产生的是随机 IO.2,看上去似乎是,某一sql查询导致读操作.结论:从以上总结出,所有的读 IOPS都是mysql进程在执行某些读查询时产生的.5.使用mysqladmin命令,来查找是哪个慢查询导致的.#./mysqladminpstrongmailprocesslist+|Id|User|Host|db|Command|Time|State|Info+|1|root|localhost|strongmail|Sleep|10|2|root|localhost|strongmail|Sleep|8|3|root|localhost|root|Query|94|Updating|updatefailuressetupdate_datasource=Ywheredatabase_id=32andupdate_datasource=Nand|14|root|localhost|Query|0|showprocesslist分析:1,MySQL数据库里,似乎在不断的运