Sunday, August 29, 2010

ORA-15042: ASM disk "5" is missing from group number "2"

On 3 nodes 11gR2 RAC, while dismounting and mounting DATA2 resulted in below error on third node that has data2 external redundancy .Alter diskgroup data2 check succeed on node1,node2 that means it does not check ASM disk headers in check command.Analyzing errors in ASM alertlog:

while mounting DATA2 error
NOTE: De-assigning number (2,0) from disk (/dev/asmr5dsk0001)
NOTE: De-assigning number (2,1) from disk (/dev/asmr5dsk0002)
NOTE: De-assigning number (2,2) from disk (/dev/asmr5dsk0003)
NOTE: De-assigning number (2,3) from disk (/dev/asmr5dsk0004)
NOTE: De-assigning number (2,4) from disk (/dev/asmr5dsk0005)
NOTE: De-assigning number (2,5) from disk ()
NOTE: De-assigning number (2,6) from disk (/dev/asmr5dsk0007)
NOTE: De-assigning number (2,7) from disk (/dev/asmr5dsk0011)
NOTE: De-assigning number (2,8) from disk (/dev/asmr5dsk0012)
ERROR: diskgroup DATA2 was not mounted
NOTE: cache deleting context for group DATA2 2/-1905567771
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "5" is missing from group number "2"
ERROR: alter diskgroup DATA2 mount
Wed Aug 25 15:10:48 2010
ASM Health Checker found 1 new failures

It look for Disk /dev/asmr5dsk0006 ,
could be ASM disk header corrupted

To verify:

$ kfed read /dev/asmr5dsk0006 text=name.txt
ERROR!!!, failed to get the oracore error message

$ dd if=/dev/asmr5dsk0006 bs=512 count=1
ةآشءح���sW1+0 records in.
1+0 records out.
$ dd if=/dev/asmr5dsk0005 bs=512 count=1
Kب4#ORCLDISK
DATA2_0004DATA2DATA2_0004��}ىذ��ض��و�)5
��}�l1+0 records in.
1+0 records out.

dd output show DATA2 ASM heading missing
Also verif by
% kfed read /dev/asmr5dsk0006 > mpa002disk5.txt
% kfed read /dev/asmr5dsk0006 > mpa010disk5.txt
% kfed read /dev/asmr5dsk0006 > mpa020disk5.txt

$ kfed read /dev/asmr5dsk0006 > mpa002disk5.txt
$ more mpa002disk5.txt
kfbh.endian: 201 ; 0x000: 0xc9
kfbh.hard: 194 ; 0x001: 0xc2
kfbh.type: 212 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 193 ; 0x003: 0xc1
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message

$ kfed read /dev/asmr5dsk0006 > mpa010disk5.txt
$ more mpa010disk5.txt
kfbh.endian: 201 ; 0x000: 0xc9
kfbh.hard: 194 ; 0x001: 0xc2
kfbh.type: 212 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 193 ; 0x003: 0xc1
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message
$ ssh mpa020

$ kfed read /dev/asmr5dsk0006 > mpa020disk5.txt
$ more mpa020disk5.txt
kfbh.endian: 201 ; 0x000: 0xc9
kfbh.hard: 194 ; 0x001: 0xc2
kfbh.type: 212 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 193 ; 0x003: 0xc1
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message


Confirm , ASM disk header is corrupted so it is not detecting at
mount. ASM always backup disk header automatically. Repair by below
command and mount the disk.

$ kfed repair /dev/asmr5dsk0006

$ kfed read /dev/asmr5dsk0006 > disk5.txt
$ kfed read /dev/asmr5dsk0006 > disk5.txt
$ cat disk5.txt
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483653 ; 0x008: TYPE=0x8 NUMB=0x5
kfbh.check: 1271477282 ; 0x00c: 0x4bc93422
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 5 ; 0x024: 0x0005
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA2_0005 ; 0x028: length=10
kfdhdb.grpname: DATA2 ; 0x048: length=5
kfdhdb.fgname: DATA2_0005 ; 0x068: length=10
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32939173 ; 0x0a8: HOUR=0x5 DAYS=0x5 MNTH=0x7 YEAR=0x7da
kfdhdb.crestmp.lo: 2112475136 ; 0x0ac: USEC=0x0 MSEC=0x274 SECS=0x1e MINS=0x1f
kfdhdb.mntstmp.hi: 32940502 ; 0x0b0: HOUR=0x16 DAYS=0xe MNTH=0x8 YEAR=0x7da
kfdhdb.mntstmp.lo: 3052791808 ; 0x0b4: USEC=0x0 MSEC=0x17a SECS=0x1f MINS=0x2d
kfdhdb.secsize: 512 ; 0x0b8: 0x0200
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize: 207157 ; 0x0c4: 0x00032935
kfdhdb.pmcnt: 3 ; 0x0c8: 0x00000003
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000
kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000
kfdhdb.dbcompat: 185597952 ; 0x0e0: 0x0b100000
kfdhdb.grpstmp.hi: 32939173 ; 0x0e4: HOUR=0x5 DAYS=0x5 MNTH=0x7 YEAR=0x7da
kfdhdb.grpstmp.lo: 2111597568 ; 0x0e8: USEC=0x0 MSEC=0x31b SECS=0x1d MINS=0x1f
kfdhdb.vfstart: 0 ; 0x0ec: 0x00000000
kfdhdb.vfend: 0 ; 0x0f0: 0x00000000
kfdhdb.spfile: 0 ; 0x0f4: 0x00000000
kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000
kfdhdb.ub4spare[0]: 0 ; 0x0fc: 0x00000000
kfdhdb.ub4spare[1]: 0 ; 0x100: 0x00000000
kfdhdb.ub4spare[2]: 0 ; 0x104: 0x00000000
kfdhdb.ub4spare[3]: 0 ; 0x108: 0x00000000
kfdhdb.ub4spare[4]: 0 ; 0x10c: 0x00000000
kfdhdb.ub4spare[5]: 0 ; 0x110: 0x00000000
kfdhdb.ub4spare[6]: 0 ; 0x114: 0x00000000
kfdhdb.ub4spare[7]: 0 ; 0x118: 0x00000000
kfdhdb.ub4spare[8]: 0 ; 0x11c: 0x00000000
kfdhdb.ub4spare[9]: 0 ; 0x120: 0x00000000
kfdhdb.ub4spare[10]: 0 ; 0x124: 0x0

No comments: