丢失的ASM磁盘组.doc

近日遇到一个问题，ASM的磁盘组无法挂载，之前是正常的，由于一些其他的操作，数据库启动失败，当问题排除时候，发现在数据库整体启动时，挂载磁盘组的环节出现问题。环境介绍#硬件：Vmware ESX虚拟机OS: Red hat linux 5Oracle version: ASM disk是通过 asmlib挂载的这个磁盘组只有一个虚拟出的硬盘，是 /dev/sdb1.#下面是我整个分析的过程1. 首先通过ASM alert.log，发现如下错误,磁盘挂载失败，无法找到磁盘组1. SQLalterdiskgroupDATAmount2. NOTE:cacheregisteredgroupDATAnumber=1incarn=0xc28a1e2d3. NOTE:cachebeganmount(first)ofgroupDATAnumber=1incarn=0xc28a1e2d4. TueDec1118:06:5520125. ERROR:noPSTquorumingroup:required2,found06. NOTE:cachedismounting(clean)group1/0xC28A1E2D(DATA)7. NOTE:dbwrnotbeingmsgdtodismount8. NOTE:lgwrnotbeingmsgdtodismount9. NOTE:cachedismountedgroup1/0xC28A1E2D(DATA)10. NOTE:cacheendingmount(fail)ofgroupDATAnumber=1incarn=0xc28a1e2d11. NOTE:cachedeletingcontextforgroupDATA1/0xc28a1e2d12. GMONdismountinggroup1at8forpid17,osid3216313. ERROR:diskgroupDATAwasnotmounted14. ORA-15032:notallalterationsperformed15. ORA-15017:diskgroupDATAcannotbemounted16. ORA-15063:ASMdiscoveredaninsufficientnumberofdisksfordiskgroupDATA17. ERROR:alterdiskgroupDATAmount 2. 首先检查ASM pfile 文件，未发现异常1. asm_diskgroups=DATA2. instance_type=asm3. large_pool_size=12M4. remote_login_passwordfile=EXCLUSIVE3. 尝试通过以下命令检查磁盘是否物理存在，是如何对应物理设备的，发现查询不到ASM磁盘1. gridlgto_test$kfoddisks=all2. 3. -Nonoutput-4. 5. 6. 7. gridlgto_testpeer$cd/dev/oracleasm/disks/8. gridlgto_testdisks$ls9. 10. -Nonoutput-11. 12. 13. gridlgto_testdisks$/etc/init.d/oracleasmlistdisks14. 15. -Nonoutput- 4.但是直接检查物理设备，/dev/sdb1是存在的，说明OS已经识别该硬盘设备，只是ASMlib无法正常识别:1. 查询对应的物理硬盘2. oracleOEL $ /etc/init.d/oracleasm querydisk -d disk1 Disk DISK1 is a valid ASM disk on device 8,173. oracleOEL $ ls -l /dev/ |grep 8|grep 17 brw-r- 1 root disk 8, 17 Oct 16 14:01 sdb14. 5. rootlgto_test#ls-lst/dev/sd*6. 0brw-r-1rootdisk8,0Dec1119:29/dev/sda7. 0brw-r-1rootdisk8,2Dec1119:29/dev/sda28. 0brw-r-1rootdisk8,16Dec1119:29/dev/sdb9. 0brw-r-1rootdisk8,17Dec1119:29/dev/sdb1Thisisthemisseddiskgroup10. 0brw-r-1rootdisk8,1Dec1111:29/dev/sda15.起先是考虑是否是磁盘头损害，导致无法ASMlib识别该磁盘， dump磁盘头发现没有问题.1. #od-c/dev/sdb12. 3. 0000040ORCLDISKDATADG014. 5. 7760040ORCLDISKDATADG01这里补充下，如果磁盘头信息丢失，将会显示如下1. 0000040ORCLDISK00000000如果显示这个结果，需要通过以下方式renamedisk，具体可以参考文档Oracleasm Listdisks Cannot See Disks (Doc ID 392527.1)1. Usetheoracleasmrenamediskutilitytoaddanasmliblabeltothedisk:2. /etc/init.d/oracleasmrenamedisk/dev/3. Ifitfails,usethe-fswitch:4. /etc/init.d/oracleasmrenamedisk-f/dev/ 6.重启ASMLib ，检查是否是ASMLib 问题1. rootlgto_test#/etc/init.d/oracleasmrestart2. DroppingOracleASMLibdisks:3. OK4. ShuttingdowntheOracleASMLibdriver:FAILED检查文件系统oracleasm文件系统已经成功挂载1. rootlgto_test#df-ha2. FilesystemSizeUsedAvailUse%Mountedon3. 4. oracleasmfs000-/dev/oracleasm7.检查 /dev/sdb1状态，查看是否已经marked为ASM disk，显示已经标记成功1. rootlgto_test#oracleasmquerydisk/dev/sdb12. Device/dev/sdb1ismarkedanASMdiskwiththelabelDATADG013. 4. rootlgto_test#/sbin/serviceoracleasmscandisks5. ScanningthesystemforOracleASMLibdisks:6. OK7. 8. rootlgto_test#/etc/init.d/oracleasmlistdisks9. -Nonoutput-8.检查 rpm package也没有问题1. gridlgto_test$rpm-qa|greporacleasm2. oracleasmlib-2.0.4-1.el53. oracleasm-support-2.1.7-1.el54. oracleasm-2.6.18-308.el5-2.0.5-1.el5 9. 收集 Kfed logs,没有检查到异常新信息。1. rootlgto_test#/oracle/ora11g/product/app/grid/bin/kfedread/dev/sdb12. 3. kfbh.endian:1;0x000:0x014. 5. kfbh.hard:130;0x001:0x826. 7. kfbh.type:1;0x002:KFBTYP_DISKHEAD8. 9. kfbh.datfmt:1;0x003:0x0110. 11. kfbh.block.blk:0;0x004:T=0NUMB=0x012. 13. kfbh.block.obj:2147483648;0x008:TYPE=0x8NUMB=0x014. 15. kfbh.check:3351358462;0x00c:0xc7c1abfe16. 17. kfbh.fcn.base:0;0x010:0x0000000018. 19. kfbh.fcn.wrap:0;0x014:0x0000000020. 21. kfbh.spare1:0;0x018:0x0000000022. 23. kfbh.spare2:0;0x01c:0x0000000024. 25. vstr:ORCLDISKDATADG01;0x000:length=1626. 27. kfdhdb.driver.reserved0:1096040772;0x008:0x4154414428. 29. kfdhdb.driver.reserved1:825247556;0x00c:0x3130474430. 31. kfdhdb.driver.reserved2:0;0x010:0x0000000032. 33. kfdhdb.driver.reserved3:0;0x014:0x0000000034. 35. kfdhdb.driver.reserved4:0;0x018:0x0000000036. 37. kfdhdb.driver.reserved5:0;0x01c:0x0000000038. 39. pat:186646528;0x020:0x0b20000040. 41. kfdhdb.dsknum:0;0x024:0x000042. 43. kfdhdb.grptyp:1;0x026:KFDGTP_EXTERNAL44. 45. kfdhdb.hdrsts:3;0x027:KFDHDR_MEMBER46. 47. kfdhdb.dskname:DATADG01;0x028:length=848. 49. kfdhdb.grpname:DATA;0x048:length=450. 51. kfdhdb.fgname:DATADG01;0x068:length=852. 53. kfdhdb.capname:;0x088:length=054. 55. kfdhdb.crestmp.hi:32977140;0x0a8:HOUR=0x14DAYS=0x7MNTH=0xcYEAR=0x7dc56. 57. kfdhdb.crestmp.lo:1642529792;0x0ac:USEC=0x0MSEC=0x1c1SECS=0x1eMINS=0x1858. 59. kfdhdb.mntstmp.hi:32977140;0x0b0:HOUR=0x14DAYS=0x7MNTH=0xcYEAR=0x7dc60. 61. kfdhdb.mntstmp.lo:1664549888;0x0b4:USEC=0x0MSEC=0x1c1SECS=0x33MINS=0x1862. 63. kfdhdb.secsize:512;0x0b8:0x020064. 65. kfdhdb.blksize:4096;0x0ba:0x100066. 67. kfdhdb.ausize:1048576;0x0bc:0x0010000068. 69. kfdhdb.mfact:113792;0x0c0:0x0001bc8070. 71. kfdhdb.dsksize:204797;0x0c4:0x00031ffd72. 73. kfdhdb.pmcnt:3;0x0c8:0x0000000374. 75. kfdhdb.fstlocn:1;0x0cc:0x0000000176. 77. kfdhdb.altlocn:2;0x0d0:0x0000000278. 79. kfdhdb.f1b1locn:2;0x0d4:0x0000000280. 81. kfdhdb.redomirrors0:0;0x0d8:0x000082. 83. kfdhdb.redomirrors1:0;0x0da:0x000084. 85. kfdhdb.redomirrors2:0;0x0dc:0x000086. 87. kfdhdb.redomirrors3:0;0x0de:0x000088. 89. kfdhdb.dbcompat:168820736;0x0e0:0x0a10000090. 91. kfdhdb.grpstmp.hi:32977140;0x0e4:HOUR=0x14DAYS=0x7MNTH=0xcYEAR=0x7dc92. 93. kfdhdb.grpstmp.lo:1642390528;0x0e8:USEC=0x0MSEC=0x139SECS=0x1eMINS=0x1894. 95. kfdhdb.vfstart:0;0x0ec:0x0000000096. 97. kfdhdb.vfend:0;0x0f0:0x0000000098. 99. kfdhdb.spfile:58;0x0f4:0x0000003a100. 101. kfdhdb.spfflg:1;0x0f8:0x00000001102. 103. kfdhdb.ub4spare0:0;0x0fc:0x00000000104. 105. kfdhdb.ub4spare1:0;0x100:0x00000000106. 107. kfdhdb.ub4spare2:0;0x104:0x00000000108. 109. kfdhdb.ub4spare3:0;0x108:0x00000000110. 111. kfdhdb.ub4spare4:0;0x10c:0x00000000112. 113. kfdhdb.ub4spare5:0;0x110:0x00000000114. 115. kfdhdb.ub4spare6:0;0x114:0x00000000116. 117. kfdhdb.ub4spare7:0;0x118:0x00000000118. 119. kfdhdb.ub4spare8:0;0x11c:0x00000000120. 121. kfdhdb.ub4spare9:0;0x120:0x00000000122. 123. kfdhdb.ub4spare10:0;0x124:0x00000000124. 125. kfdhdb.ub4spare11:0;0x128:0x00000000126. 127. kfdhdb.ub4spare12:0;0x12c:0x00000000128. 129. kfdhdb.ub4spare13:0;0x130:0x00000000130. 131. kfdhdb.ub4spare14:0;0x134:0x00000000132. 133. kfdhdb.ub4spare15:0;0x138:0x00000000134. 135. kfdhdb.ub4spare16:0;0x13c:0x00000000136. 137. kfdhdb.ub4spare17:0;0x140:0x00000000138. 139. kfdhdb.ub4spare18:0;0x144:0x00000000140. 141. kfdhdb.ub4spare19:0;0x148:0x00000000142. 143. kfdhdb.ub4spare20:0;0x14c:0x00000000144. 145. kfdhdb.ub4spare21:0;0x150:0x00000000146. 147. kfdhdb.ub4spare22:0;0x154:0x00000000148. 149. kfdhdb.ub4spare23:0;0x158:0x00000000150. 151. kfdhdb.ub4spare24:0;0x15c:0x00000000152. 153. kfdhdb.ub4spare25:0;0x160:0x00000000154. 155. kfdhdb.ub4spare26:0;0x164:0x00000000156. 157. kfdhdb.ub4spare27:0;0x168:0x00000000158. 159. kfdhdb.ub4spare28:0;0x16c:0x00000000160. 161. kfdhdb.ub4spare29:0;0x170:0x00000000162. 163. kfdhdb.ub4spare30:0;0x174:0x00000000164. 165. kfdhdb.ub4spare31:0;0x178:0x00000000166. 167. kfdhdb.ub4spare32:0;0x17c:0x00000000168. 169. kfdhdb.ub4spare33:0;0x180:0x00000000170. 171. kfdhdb.ub4spare34:0;0x184:0x00000000172. 173. kfdhdb.ub4spare35:0;0x188:0x00000000174. 175. kfdhdb.ub4spare36:0;0x18c:0x00000000176. 177. kfdhdb.ub4spare37:0;0x190:0x00000000178. 179. kfdhdb.ub4spare38:0;0x194:0x00000000180. 181. kfdhdb.ub4spare39:0;0x198:0x00000000182. 183. kfdhdb.ub4spare40:0;0x19c:0x00000000184. 185. kfdhdb.ub4spare41:0;0x1a0:0x00000000186. 187. kfdhdb.ub4spare42:0;0x1a4:0x00000000188. 189. kfdhdb.ub4spare43:0;0x1a8:0x00000000190. 191. kfdhdb.ub4spare44:0;0x1ac:0x00000000192. 193. kfdhdb.ub4spare45:0;0x1b0:0x00000000194. 195. kfdhdb.ub4spare46:0;0x1b4:0x00000000196. 197. kfdhdb.ub4spare47:0;0x1b8:0x00000000198. 199. kfdhdb.ub4spare48:0;0x1bc:0x00000000200. 201. kfdhdb.ub4spare49:0;0x1c0:0x00000000202. 203. kfdhdb.ub4spare50:0;0x1c4:0x00000000204. 205. kfdhdb.ub4spare51:0;0x1c8:0x00000000206. 207. kfdhdb.ub4spare52:0;0x1cc:0x00000000208. 209. kfdhdb.ub4spare53:0;0x1d0:0x00000000210. 211. kfdhdb.acdb.aba.seq:0;0x1d4:0x00000000212. 213. kfdhdb.acdb.aba.blk:0;0x1d8:0x00000000214. 215. kfdhdb.acdb.ents:0;0x1dc:0x0000216. 217. kfdhdb.acdb.ub2spare:0;0x1de:0x000010. 阶段总结，通过以上的分析，得出以下总结1. ASMLib 正常2. RPM包正常3. 磁盘头没有损坏和丢失信息4. 该硬件已经被系统正常识别当前问题就是为什么ASMLib 不能正常扫描并识别到该硬盘11. 最后在检查文件 /etc/sysconfig/oracleasm时，发现问题，