PowerPath Best Practice for AIX v7.5

1、 PowerPath Best Practice for AIX2008 年年 8 月月EMC Global Service TIS版本记录版本记录版本号版本号版本日期版本日期修改者修改者说明说明0.12006 年 9 月 13 日王进创建 PowerPath Best Practice for AIX 模板0.12006 年 10 月 8 日夏卫东添加内容2.02006 年 11 月 22 日夏卫东添加项目 2.3, 3.3, 3.4, 3.5, 3.6, 3.7, 4.23.02007 年 1 月 2 日夏卫东在版本记录中标明添加项目；添加项目 3.8, 3.9, 4.34.02007 年

2、 1 月 27 日王进增加一个章节 4.4: emc1081275.02007 年 2 月 28 日夏卫东增加 AIXV5 patch level 信息，4GB 光纤卡信息，增加一个章节 4.5：emc1116356.02007 年 4 月 2 日夏卫东增加 1.2：PP5.0 AIX requirement；增加3.10：如何修改 VCMDB 使不同操作系统共享FA ports；更新 2.2: EMC69100 内容6.12007 年 4 月 16 日夏卫东增加 3.12：关于 AIX4.3.3 的内容7.02007 年 5 月 28 日夏卫东增加 3.11: TF/Clone experi

3、ence sharing4.6：use PowerPath instead of AIX MPIO4.7：emc116729 FSCSI_ERR104.8: emc155602 iostat incorrect7.12008 年 1 月 6 日李晓川陈铭更新 AIX 的版本增加 4.9：EMC163961: AIX Host Lost access to storage during powermt restore( same as powermt coinfig / syminq / symrdf / symsnap)7.22008 年 2 月 14 日陈铭更新 AIX 的版本增加 4 po

4、werpath 5.1 和 SP1 对相关问题的改进7.32008 年 4 月 26 日陈铭更新 AIX 的版本增加 3.10 关于 Symmetrix 锁的相关案例7.42008 年 6 月 2 日陈铭针对 HACMP5.4 再次更新 2.2: EMC69100 内容删除 2.3 Starting from HACMP5.4, Primus69100 “emcpowerreset” not needed添加 2.3 EMC156011 推荐在 HACMP5.4 继续使用 emcpowerreset7.52008 年 8 月 12 日陈铭增加 3.11 emc81931 emc81930 通过

5、Solution Enabler 清除存储的 SCSI Reservation Lock版权说明版权说明本文件中出现的任何文字叙述、文档格式、插图、照片、方法、过程等内容，除另有特别注明，版权均属美国EMC 公司所有，受到有关产权及版权法保护。任何个人、机构未经美国 EMC 公司的书面授权许可，不得复制、引用或传播本文件的任何片断，无论通过电子形式或非电子形式。目目 录录1AIX HOST CONNECTIVITY.61.1DOWNLOADING LATEST AIX ODM FROM EMC WEBSITE:.61.2AIX FIX LEVEL INFORMATION:.61.3FC ADA

6、PTER USEFUL COMMANDS:.81.4USEFUL SETTING AND PARAMETERS:.121.4.1FSCSI device Fast_Fail & Dyntrk.121.4.2Set HACMP “break reserves in parallel =true” (Recommended).121.5USEFUL SMALL SCRIPTS: .121.5.1删除所有 hdiskpower 设备：.121.5.2删除从 hdiskpower0 到 hdiskpower100 的设备： .131.5.3删除所有 hdisk 设备：.131.5.4删除某块光

7、纤卡的 hdisk 设备.132HACMP.142.1HACMP 切换非常慢，每个 EMC 设备的切换时间是 2-3 分钟。 .142.2EMC 69100: HACMP 手工的TAKEOVER切换成功，但是当一台主机宕机时，EMC 硬盘的切换失败，HACMP 无法清除硬盘的 SCSI RESERVATION状态。 .172.3EMC 建议在 HACMP5.4 中继续使用EMCPOWERRESET.272.4TOPAS 或 IOSTAT 显示 EMC 硬盘 100%忙，但是实际的 I/O 却非常小或者没有。.292.5即使重启也无法删除HDISKPOWER设备,系统始终显示设备 BUSY.29

8、2.6EMC143075, “SET_SCSI_ID” HAS BEEN REPLACED BY “CFGSCSI_ID” WITH CLARIION LUNS IN HACMP/PP ENVIRONMENT.292.7CLARIION LUN 在 AIX 操作系统上的识别顺序总结-AUTHOR： DANNY LI.332.8AIX CHDEV COMMAND CAUSE DISK OPERATION ERROR ON CLARIION PASSIVE(STANDBY) PATH - AUTHOR ：DANNY LI.342.9关于 QUEUE DEPTH 参数-AUTHOR： DANNY L

9、I.372.10AIX JFS2 MOUNT OPTION CIO CAUSE VERY LOW PERFORMANCE FOR CP AND DD COMMAND- AUTHOR： DANNY LI.382.11EMC PROCEDURES WHEN REPLACING HBA CARD ON AIX- BY DANNY LI.392.12STEPS FOR TF/SNAP IMPLEMENTATION - BY DANNY LI.392.13如何修改 VCMDB 使不同操作系统共享 FA PORTS:.402.14COOKBOOK FOR TIMEFINDER INCREMENTAL CL

10、ONE AND CLONE RESTORE.402.15AIX433 环境下的技术问题.403POWERPATH .403.1FAST I/O FAILURE AND DYNAMIC TRACKING.413.2EMC145919: POWERPATH 4.5.X: AIX LVM I/O ERRORS DUE TO MISSING DEVICE RESERVATIONS.453.3SUGGEST NOT TO CHANGE “FC_ERR_RECOV” DEFAULT VALUE.473.4EMC108127: METHOD ERROR /ETC/METHODS/CFGPOWER -L PO

11、WERPATH0 0514-040 ERROR INITIALIZING A DEVICE INTO THE KERNEL IN AIX WITH MINOR NUMBER MISMATCH.473.5EMC111635: HOW TO REMOVE LUNZ DEVICES ON AN AIX HOST.503.6USE POWERPATH INSTEAD OF AIX MPIO.523.7EMC116729: FSCSI_ERR10 CONFIGURATION MISMATCH ERRORS IN ERRPT AFTER TURNING ON DYNAMIC TRACKING.533.8E

12、MC155602: AIX HOST WITH POWERPATH DOESNT UPDATE DKSTAT CORRECTLY THUS IOSTAT SUMMARY APPEARS INCORRECT.553.9EMC163961: AIX HOST LOST ACCESS TO STORAGE DURING POWERMT RESTORE( SAME AS POWERMT COINFIG / SYMINQ / SYMRDF / SYMSNAP).553.10关于 SYMMETRIX锁的相关案例.563.11通过 SOLUTION ENABLER 清除存储的 SCSI RESERVATIO

13、N LOCK.581AIX Host Connectivity1.1Downloading latest AIX ODM from EMC website:ftp:/ FIX level information:PowerPath5.0 AIX Maintenance Level requiremnt：AIX 6.1 service pack:NameTypePrereqsDate6100-00-04-0815 Service Pack ? 2008 6100-00-03-0808 Service Pack ? 2008 6100-00-02-0750 Service Pack ? 2007

14、6100-00-01-0748 Service Pack ? 2007 AIX 5.3 service pack:NameTypePrereqsDate5300-07-03-0811 Service Pack 5300-07-00-0747 ? 2008 5300-07-02-0806 Service Pack 5300-07-00-0747 ? 2008 5300-07-01-0748 Service Pack 5300-07-00-0747 ? 2007 5300-07-00-0747 Technology Level ? 2007 5300-06-06-0811 Service Pack

15、 5300-06 ? 2008 5300-06-05-0806 Service Pack 5300-06 ? 2008 5300-06-04-0748 Service Pack 5300-06 ? 2007 5300-06-03-0732 Service Pack 5300-06 ? 2007 5300-06-02-0727 Service Pack 5300-06 ? 2007 5300-06-01-0722 Service Pack 5300-06 ? 2007 5300-06 Technology Level ? 2007 5300-05-CSP Concluding Service P

16、ack 5300-05 ? 2007 5300-05-06 Service Pack 5300-05 ? 2007 5300-05-05 Service Pack 5300-05 ? 2007 5300-05-04 Service Pack 5300-05 ? 2006 5300-05-03 Service Pack 5300-05 ? 2006 5300-05-02 Service Pack 5300-05 ? 2006 5300-05-01 Service Pack 5300-05 ? 2006 5300-05 Technology Level ? 2006 5300-04-CSP Con

17、cluding Service Pack 5300-04 ? 2006 5300-04-03 Service Pack 5300-04 ? 2006 5300-04-02 Service Pack 5300-04 ? 2006 5300-04-01 Service Pack 5300-04 ? 2006 5300-04 Technology Level ? 2006 5300-03 Maintenance Level ? 2005 5300-02 Maintenance Level ? 2005 5300-01 Maintenance Level ? 2005 AIX5.2 service p

18、ack:Technology Levels and Service PacksNameTypePrereqsDate5200-10-04-0750Service PackTL 5200-10December 20075200-10-03-0744Service PackTL 5200-10October 20075200-10-02-0730Service PackTL 5200-10July 20075200-10-01-0722Service PackTL 5200-10June 20075200-10Technology Level June 20075200-09-CSPConclud

19、ing Service PackTL 5200-09June 20075200-09-06Service PackTL 5200-09March 20075200-09-05Service PackTL 5200-09February 20075200-09-04Service PackTL 5200-09January 20075200-09-03Service PackTL 5200-09November 20065200-09-02Service PackTL 5200-09October 20065200-09-01Service PackTL 5200-09September 200

20、65200-09Technology Level August 20065200-08-CSPConcluding Service PackTL 5200-08August 20065200-08-02Service PackTL 5200-08April 20065200-08-01Service PackTL 5200-08February 20065200-08Technology Level February 20065200-07-CSPConcluding Service Pack5200-07March 2006Maintenance Level packages (legacy

21、)NameTypePrereqsDate5200-07Maintenance Level September 20055200-06Maintenance Level May 20055200-05Maintenance Level January 20055200-04Maintenance Level December 20045200-03Maintenance Level May 20045200-02Maintenance Level October 20035200-01Maintenance Level May 2003AIX 5.1 service pack:ML5100-09

22、September, 2005 SW fixesML5100-08May, 2005 SW fixesML5100-07October, 2004 SW fixes1.3FC adapter useful commands:#lsdev -Cc adapter|grep fcs 查找系统所安装的光纤卡查找系统所安装的光纤卡fcs0 Available 20-58 FC Adapterfcs1 Available 20-60 FC Adapter#lscfg -vl fcsX查看光纤卡硬件信息，例如查看光纤卡硬件信息，例如:卡的类型卡的类型, WWN, 微码版本微码版本Where X is th

23、e number of the adapter returned by the previous lsdev command.4GB single port Fiber channel card:FC5758 ( 卡上标签为卡上标签为 280D,1905 )fcs3 P1-I1/Q1 FC AdapterPart Number.03N5014EC Level.ASerial Number.1E313BB001Feature Code/Marketing ID.280DManufacturer.001EFRU Number. 03N5014Device Specific.(ZM).3Networ

24、k Address.10000000C9488F4C （FC card WWN）ROS Level and ID.02C82135Device Specific.(Z0).1036406DDevice Specific.(Z1).00000000Device Specific.(Z2).00000000Device Specific.(Z3).03000909Device Specific.(Z4).FFC01158Device Specific.(Z5).02C82135Device Specific.(Z6).06C32135Device Specific.(Z7).07C32135Dev

25、ice Specific.(Z8).20000000C9488F4CDevice Specific.(Z9).BS2.10X8 =微码版本：微码版本：2.10X8Device Specific.(ZA).B1D2.10X8 Device Specific.(ZB).B2D2.10X8 Device Specific.(YL).P1-I1/Q14GB dual port Fiber channel card:FC5759 ( 卡上标签为卡上标签为 5759,1910 )fcs7 P1-I2/Q1 FC Adapter Part Number.03N5029 EC Level.A Serial N

26、umber.1F5310C048 Manufacturer.001F Feature Code/Marketing ID.5759 FRU Number. 03N5029 Device Specific.(ZM).3 Network Address.10000000C94ABA08 (FC card WWN) ROS Level and ID.02C82132 Device Specific.(Z0).1036406D Device Specific.(Z1).00000000 Device Specific.(Z2).00000000 Device Specific.(Z3).0300090

27、9 Device Specific.(Z4).FFC01155 Device Specific.(Z5).02C82132 Device Specific.(Z6).06C12132 Device Specific.(Z7).07C12132 Device Specific.(Z8).20000000C94ABA08 Device Specific.(Z9).BS2.10X8 = firmware level Device Specific.(ZA).B1F2.10X2 Device Specific.(ZB).B2F2.10X2 Device Specific.(YL).P1-I2/Q12G

28、B single port Fiber card: FC5716 ( 卡上标签为卡上标签为 280B,197E,1957 ) 常用于常用于 P5P5 系列主机上：系列主机上：fcs1 P1-C1-T1 FC Adapter Part Number.80P4543 EC Level.A Serial Number.1A31600450 Manufacturer.001A Feature Code.280B 或 197E 或 1957 FRU Number. 80P4544 Device Specific.(ZM).3 Network Address.10000000C9332A79 (FC ca

29、rd WWN) ROS Level and ID.02881914 Device Specific.(Z0).1001206D Device Specific.(Z1).00000000 Device Specific.(Z2).00000000 Device Specific.(Z3).03000909 Device Specific.(Z4).FF801315 Device Specific.(Z5).02881914 Device Specific.(Z6).06831914 Device Specific.(Z7).07831914 Device Specific.(Z8).20000

30、000C940F419 Device Specific.(Z9).TS1.90A4 = firmware level Device Specific.(ZA).T1D1.90A4 Device Specific.(ZB).T2D1.90A4 Device Specific.(YL).U7879.001.DQD0JHD-P1-C1-T1 2GB single port Fiber card:FC6239（卡上标签为（卡上标签为 5704） 常用于常用于 P6XX 系列主机上：系列主机上：fvs0 3F-08 FC Adapter Part Number.80P6415 EC Level.A Se

31、rial Number.1D51406697 Manufacturer.001D Feature Code/Marketing ID.5704 FRU Number. 80P6416 Device Specific.(ZM).3 Network Address.10000000C9492EF6 ROS Level and ID.02E01971 Device Specific.(Z0).2003806D Device Specific.(Z1).00000000 Device Specific.(Z2).00000000 Device Specific.(Z3).03000909 Device

32、 Specific.(Z4).FF601410 Device Specific.(Z5).02E01971 Device Specific.(Z6).06631971 Device Specific.(Z7).07631971 Device Specific.(Z8).20000000C9492EF6 Device Specific.(Z9).HS1.91X1 = firmware level Device Specific.(ZA).H1D1.91X1 Device Specific.(ZB).H2D1.91X1 Device Specific.(YL).P1-I5/Q1FC6228 比较老

33、型号的光纤卡，比较老型号的光纤卡，2GB 速率速率：fcs0 00-04 IBM Gigabit Fiber Channel PCI Adapter for 64 bit busPart Number.00P4494EC Level.ASerial Number.1A25001206Manufacturer.001AFRU Number. 00P4495Network Address.10000000C930D02BROS Level and ID.02C03951Device Specific.(Z0).2002606DDevice Specific.(Z1).00000000Device

34、Specific.(Z2).00000000Device Specific.(Z3).03000909Device Specific.(Z4).FF401210Device Specific.(Z5).02C03951Device Specific.(Z6).06433951Device Specific.(Z7).07433951Device Specific.(Z8).20000000C930D02BDevice Specific.(Z9).CS3.91X4 Device Specific.(ZA).C1D3.91X4 Device Specific.(ZB).C2D3.91X4FC622

35、7老型号光纤卡，老型号光纤卡，1GB 速率：速率：fcs0 00-04 IBM Gigabit Fiber Channel PCI Adapter ROS Level and ID.02903291 Device Specific (Z9).SS3.22A1 Part Number.09P4038, 09P1162, 03N4167, 24L0023 Serial Number.00000000 Device Specific.(YL).P2-I5#lsattr -El fscsi0查看查看 fscsi 设备的连接状态和设备的连接状态和 fastfail, dyntrk 等参数设置等参数设置s

36、csi_id 0 x10300 Adapter SCSI ID Falseattach switch How this adapter is CONNECTED Falsesw_fc_class 3 FC Class for Fabric Truefc_err_recov delayed_fail FC Fabric Event Error RECOVERY Policy True#rmdev -dl fscsi0 删除光纤协议设备，不建议删除删除光纤协议设备，不建议删除 fcs0 设备。设备。#rmdev -dl fscsi0 -R 删除光纤卡及其所连接设备，例如删除光纤卡及其所连接设备，例

37、如 hdiskpower, hdisk, FC tape#cfgmgr -vl fcs0配置某光纤卡连接设备配置某光纤卡连接设备#cfgmgr -v配置所有设备配置所有设备#/usr/lpp/EMC/Symmetrix/bin/emc_cfgmgr 配置配置 EMC Hdisk 设备设备, 尤其是设备尤其是设备如果某台主机只访问如果某台主机只访问 BCV 设备，需要在设备，需要在/etc/inittab 文件结尾增加一行命令：文件结尾增加一行命令：$/mkbcv -a。这样当该主机重启时，。这样当该主机重启时，BCV 设备会自动变为设备会自动变为Available 状态。状态。1.4Usefu

38、l Setting and Parameters:1.4.1FSCSI device Fast_Fail & Dyntrk 相关说明参见 2.1& 4.3：chdev -l fscsi0 -a fc_err_recov=fast_fail -Pchdev -l fscsi0 -a dyntrk=yes -PNote: Only enable it in “multipath software”(such as PowerPath) environment. Only enable it in Switch environment.1.4.2Set HACMP “break re

39、serves in parallel =true” (Recommended)相关说明参见 4.1 ：smitty hacmp - extended configuration - extended resource configuration - HACMP Extended resources configuration - configure custom disk methods - change/show Custom disk methods -select disk/pseudo/power then scroll down to break reserves in parall

40、el and change the value to true.1.5useful small scripts:1.5.1删除所有删除所有 hdiskpower 设备：设备： for i in lsdev -Cc disk|grep hdiskpower|cut -c 1-13dormdev -dl $idone1.5.2删除从删除从 hdiskpower0 到到 hdiskpower100 的设备：的设备：# i=0# while i -le 100 dormdev -dl hdiskpower$ilet i=i+1done1.5.3删除所有删除所有 hdisk 设备：设备：# for i

41、in lsdev -Cc disk|grep EMC|cut -c 1-8dormdev -dl $idone1.5.4删除某块光纤卡的删除某块光纤卡的 hdisk 设备设备# lsdev -Cc adapter|grep fcs0fcs0 Available 20-58 FC Adapter# for i in lsdev -Cc disk|grep 20-58|cut -c 1-8dormdev -dl $idone2HACMP2.1 HACMP 切换非常慢，每个切换非常慢，每个 EMC 设备的切换时间是设备的切换时间是 2-3 分钟。分钟。Answer 1: set break rese

42、rves in parallel = trueID: emc122557Domain: EMC1Solution Class: 3.X CompatibilityGoal How do I set up break reserves in parallel on AIX HACMP? Fact OS: IBM AIX 5.xFact Application SW: HACMP 5.XFact break reserves in parallel = falseSymptom HACMP Cluster failover is slowFix set break reserves in para

43、llel = trueNote To change this setting enter: smitty hacmp - extended configuration - extended resource configuration - HACMP Extended resources configuration - configure custom disk methods - change/show Custom disk methods select disk/pseudo/power then scroll down to break reserves in parallel and

44、 change the value to true. Repeat this on all nodes in the cluster, then resync the cluster.Note: This will break reserves for all disks in each volume group in parallel. Thus a cluster with a large number of disks in each volume group will see substantially better improvement than a cluster where e

45、ach volume group contains a small number of disks.Note If disk/pseudo/power is not defined as a custom disk method on your cluster and you have PowerPath installed, please see EMC69100Answer 2: “set_scsi_id” is not installed in CLARiiON environmentID: emc97234Domain: EMC1Solution Class: 3.X Compatib

46、ilityGoal New set_scsi_id script is now required with CLARiiON LUNs in an HACMP / PP environment.Goal How to set up HACMP on AIX with CLARiiON LUNs under PowerPath control.Goal Example of the script set_scsi_id.sh needed with the PowerPath emcpowerreset utility.Fact Powerpath implemented with HACMP

47、and emcpowerreset.Fact EMC SW: PowerPath 4.3 and aboveFact Application SW: HACMPFact emcpowerreset utility usedFact Product: CLARiiON Storage ArraySymptom HACMP Cluster failover is slowSymptom emcpowerreset gets called on cluster initial start with no devices reserved.Symptom emcpowerreset gets call

48、ed during node failure event for devices with no disk reservationsSymptom emcpowerreset takes a longer period of time to execute.Cause The scdiskutil binary uses the PowerPath pseudo devices scsi_id ODM attribute but sends ioctl(s) directly to the adapter/HBA interface instead of using the PowerPath

49、 interface. In some cases this scsi_id ODM attribute will correspond to the CLARiiON passive not ready?path rather then the Active?path to the device required. When HACMP tries to access a LUN down the passive not ready?path, the following problems can be encountered:1) In a Cascading HACMP env, the

50、 HACMP failover time will increase for each device accessed down the passive path.2) In a Concurrent Access env, the emcpowerreset?utility might get run on some devices down the passive path resulting in a Permanent SCSI Reservation on LUNs which should never have SCSI Reservation. See Primus case e

51、mc104555 for more details on this issue.Fix Fixed in PP 4.6 (ETA Q3/2006). In the mean time Customer can get the fix from EMCs PowerLink web site.All of the above problems have been fixed with a new version on the set_scsi_id?script. To address a number of backward and forward compatibility issues,

52、this new script now combines the original set_scsi_id.sh script with part of the existing rc.emcpower.Also note:1.This new set_scsi_id?script is now required for ALL CLARiiON attach devices to an AIX node with HACMP and PP. 2.This new set_scsi_id?script will replace the existing versions of the set_

53、scsi_id.sh?and rc.emcpower?scripts. 3.With this new set_scsi_id?script the old rc.emcpower utility is no longer required. 4.This new set_scsi_id?script is backward compatible with PP 4.3 and higher. 5.The name of the file has changed from set_scsi_id.sh?to set_scsi_id? 6.The default location that th

54、e file will be installed in when added via a PP install CD has changed. It was placed in ?usr/lpp/EMC/CLARiiON/bin? it will now be put in /etc.Customers adding this new set_scsi_id?script to an AIX node for the first time should follow the steps below to acquire the file from EMCs PowerLink website

55、and then use the claddcustom command, as shown in the procedure below, to add the set_scsi_id?script into their HACMP environment.Customers who already have the previous version of the set_scsi_id.sh?script configured in their HACMP env will need to delete the old set_scsi_id.sh?script and modify th

56、eir HACMP env to point to the new file in the new location. This can be done a number of ways. EMC recommends following the exact same steps in the procedure below except, in Step 3, replace claddcustom? with clchcustom ?to change the current HACMP configuration to point to the new file name in the

57、new location.1. Copy the set_scsi_id.zip?file from EMCs PowerLink web site to a windows based system with WinZip. set_scsi_id.zip or http:/ Select Support - Downloads and Patches - Downloads D-R - PowerPath for UNIX. Then select the file set_scsi_id.zip?and follow the pop up to save the file to disk

58、.2. Uncompress the file set_scsi_id.zip?using unzip.3. Copy the script set_scsi_id?to the /etc directory of all AIX node needing it and make it executable. chmod +x /etc/set_scsi_id4. Add the custom cluster event to your configuration. This event is the name given to the script that will be added la

59、ter to select pre-defined HACMP events. Note that is is one long command./usr/es/sbin/cluster/utilities/claddcustom -t event -nset_scsi_id -ISet correct scsi id on EMC CLARiiON pseudo devices.-v/etc/set_scsi_id5. Verify your custom cluster event was added.(0)odmget HACMPcustomHACMPcustom: name = set

60、_scsi_id type = event description = Set correct scsi id on EMC CLARiiON pseudo devices. value = /etc/set_scsi_id relation = status = 06. Modify the pre-defined HACMP event by giving the event command your custom cluster event as a pre-event command./usr/es/sbin/cluster/utilities/clchevent -Onode_up -s/usr/es/s