在多节点群集中同步emcpower设备符.doc

在多节点群集中同步emcpower设备符1) 环境介绍OS：redhat enterprise Linux 4.6 x86 Cluster:RHCS （2 nodes）多路径软件：emc powerpath 5.1 for linux Storage:EMC AX4-5 EMC CX300Ax4-5有一个LUN映射给主机，CX300有两个LUN映射给主机2) 故障描述在磁阵上配置好LUN映射后，先后重新两节点服务器。两节点都认到所映射存储单元（LUN）。运行fdisk l查看LUN在主机（OS）看到的设备名。发现两节点认到的设备名不一致。其中，node1认到emcpowera、emcpowerc、emcpowerd；node2认到emcpowera、emcpowerb、emcpowerc；根据所划分空间的大小，可知其中node1 emcpowera对应node2 emcpowera，node1 emcpowerc对应node2 emcpowerb，node1 emcpowerd对应node2 emcpowerc。由于两节点要做cluster，在群集中配置共享存储时，要求两节点对识别到的LUN要有相同的设备名。3) 分析排错Node2识别到的盘符是对的；node1有问题，不知道为何把emcpowerb搞没了。在node1上执行powermt display dev=allemcpadm getfreepseudos n 5发现node1的emcpowerb并在列表中。由于业务系统上线在即，没有更多的时间去考虑和分析。当时想到两种思路，一是删除node1上识别到的路径，重启机器看看是否能解决；二是，将node2的盘符手动修改为和node1一样。排错思路一操作：powermt remove dev=all /删除当前认到的路径powermt config /路径重认powermt display dev=allreboot问题依然存在，没有得到解决；排错思路二操作：在node2上操作emcpadm getfreepseudos /发现emcpowerd 可用；emcpadm s emcpowerc t emcpowerdemcpadm s emcpowerb t emcpowercpowermt saveReboot至此，两节点都认到emcpowera,emcpowerc,emcpowerd，问题解决。4) 结论由于node1之前做测试时，曾有emcpowerb存在过，在移走该设备后， powerpath配置数据库未能及时更新。导致emcpowerb表现为占用。后续我找了相关的文章，发现通过强制删除powerpath配置的文件方式尝试进行解决。操作步骤如下：停止powerpath服务/etc/init.d/PowerPath stop保存当前配置文件的备份# cp /etc/powermt.custom /etc/powermt.custom.old_config# cp /etc/emcp_devicesDB.dat /etc/emcp_devicesDB.dat.old_config# cp /etc/emcp_devicesDB.idx /etc/emcp_devicesDB.idx.old_config删除powerpath相关配置文件 # rm /etc/powermt.custom /etc/emcp_devicesDB.dat /etc/emcp_devicesDB.idx重启powerpath服务# /etc/init.d/PowerPath start保持powerpath配置# powermt save 5) 参考powermt用法Usage: powermt class=all|clariion|ess|hitachi|hphsx|hpxp|invista|symm powermt check force hba=|all dev=|all powermt check_registration powermt config powermt disable hba= powermt display nonvirtual dev=|all every= class=invista|all width= powermt display ports dev=|all every= width= powermt display hba_mode powermt display latency dev=|all every= width= powermt display options powermt display paths every= width= powermt display port_mode powermt display unmanaged powermt enable hba= powermt load file= powermt manage dev= | class= powermt release powermt remove force hba=|all | dev=|all powermt restore hba=|all dev=|all powermt save file= powermt set mode=active|standby hba=|all dev=|all powermt set path_latency_monitor=on|off powermt set path_latency_threshold= powermt set periodic_autorestore=on|off powermt set policy=ad|bf|co|lb|li|nr|re|rr|so|si dev=|all powermt set port_disable=true|false dev= powermt set priority= dev=|all powermt unmanage dev= | class= powermt update lun_names powermt version format=verbose|legacyEmcpadm用法Usage:Pseudo device/LU mappings:-emcpadm check_mappings -v -f emcpadm export_mappings -f emcpadm import_mappings -v -f emcpadm print_mappings -p| -f Pseudo device/LU renaming:-emcpadm getfreepseudos -n -b emcpadm getfreepseudos -n -b emcpadm getusedpseudosemcpadm renamepseudo -s -t emcpadm renamepseudo -s -t Other:-emcpadm helpemcpadm versionroot cause 1：In some cases, during installation of PowerPath and device reconfiguration, a server may skip a few emcpower devices due to devices that were removed.PowerPath keeps track of devices and makes sure that the emcpower device names remains the same regardless of the underlying Linux /dev/sd# device. Fix:（steps for powerpath 4.x）1) Make sure all I/O is stopped and all of the file systems to the array are unmounted.2) Stop PowerPath # /etc/init.d/PowerPath stop3) Make a backup copy of the current PowerPath custom file just in case # cp /etc/powermt.custom /etc/powermt.custom.old_config4) Make a backup copy of the current PowerPath config dat file.just in case # cp /etc/emcp_devicesDB.dat /etc/emcp_devicesDB.dat.old_config5) Make a backup copy of the current PowerPath config idx file.just in case # cp /etc/emcp_devicesDB.idx /etc/emcp_devicesDB.idx.old_config6) Remove the old config files # rm /etc/powermt.custom /etc/emcp_devicesDB.dat /etc/emcp_devicesDB.idx7) Remove the /etc/emc/archive directory. # rm r /etc/emc/arvhive8) Start PowerPath # /etc/init.d/PowerPath start9) Save the new configuration # powermt save In some cases with PowerPath 4.x this process will clean up the PowerPath devices but they still will not be discovered in Bus-Target-LUN order so if you are trying to synchronize emcpower device numbers between two cluster nodes it may not work.In this case it is recommended that you present the devices to the node one at a time in the order you want them to appear. root cause 2： Devices were not added to the nodes in the same orderFix:（steps for powerpath 4.x） Use the emcpadm command to change the emcpower pseudo devices to the desired names.In order to fix the discrepancy between the two nodes the emcpadm command can be used.1）Use the command below in order to determine the emcpower devices that are already in use# emcpadm getused PowerPath pseudo device names in use: Pseudo Device Name Major# Minor# emcpowera 232 0 emcpowerb 232 16 emcpowerc 232 32 emcpowerd 232 48 emcpowere 232 64 emcpowerg 232 962）Use the command below in order to determine the emcpower devices that are available# emcpadm getfree -n 5 -b emcpowera PowerPath pseudo device names not in use: Pseudo Device Name Major# Minor# emcpowerf 232 80 emcpowerh 232 112 emcpoweri 232 128 emcpowerj 232 144 emcpowerk 232 1603）Use the command below to rename a device# emcpadm rename -s emcpowerg -t emcpowerf（source:gtarget:f）4）The emcpadm getused command can now be used again to check the devices after the rename# emcpadm getusedPowerPath pseudo device names in use: Pseudo Device Name Major# Minor# emcpowera 232 0 emcpowerb 232 16 emcpower