ASM after SAN upgrade

Alert: After SAN Firmware Upgrade, ASM Diskgroups ( Using ASMLIB) Cannot Be Mounted Due To ORA-15085: ASM disk "" has inconsistent sector size.  ID 1500460.1

Applies to:

Oracle Server - Enterprise Edition - Version 10.2.0.1 to 11.2.0.4 [Release 10.2 to 11.2]
Linux OS
Haansoft Linux x86
Linux x86-64

Description

After upgrade  SAN "IBM NSeries N7950T - Data ONTAP 8.0.2P3" firmware to "BM NSeries N7950T - Data ONTAP 8.1.1" firmware, the ASM diskgroups cannot be mounted due to the next errors:

SYS@+ASM2> alter diskgroup DATA11G mount;
alter diskgroup DATA11G mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA11G" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup
"DATA11G"
ORA-15085: ASM disk "" has inconsistent sector size.
ORA-15085: ASM disk "" has inconsistent sector size.
ORA-15085: ASM disk "" has inconsistent sector size.
ORA-15085: ASM disk "" has inconsistent sector size.
ORA-15085: ASM disk "" has inconsistent sector size.

Occurrence

A) Original configuration:

1. ASM RAC 11.2.0.3 .
2. ASMLIB API installed
3. Linux 64-bit
4. IBM NSeries N7950T - Data ONTAP 8.0.2P3 (firmware upgrade)
5. Diskgroups were created on 512 bytes sector size disks.

B) The problem started on the new configuration "IBM NSeries N7950T - Data ONTAP 8.1.1":

1. ASM RAC 11.2.0.3 .
2. ASMLIB API installed
3. Linux 64-bit
4. IBM NSeries N7950T - Data ONTAP 8.1.1 (firmware upgrade)
5. Diskgroups were created on 512 bytes sector size disks.

Note 1: The problem could occur on any ASM release (10.2.0.1X to 11.2.0.X).

Note 2: The problem could occur on any ASMLIB release.

Note 3: This problem could occur with any other Storage Vendor, if the firmware upgrade modifies/changes the physical_block_size = 4096 bytes (4kb).

C) This problem occurred due to the disks are now presented with "logical_block_size" = 512 bytes and "physical_block_size" = 4096 bytes (4kb) due to the SAN firmware upgrade follows:

 # cd /sys/block/dm-21/queue
 # more *block_size

 ::::::::::::::
 logical_block_size
 ::::::::::::::
 512
 ::::::::::::::
 physical_block_size
 ::::::::::::::
 4096

In other words, the logical sector size is presented = 512 bytes and the physical sector size is presented = 4096 bytes (4kb), this is not supported by ASMLIB API at this moment.

E) Despite ASM diskgroups were created on 512 bytes sector size disks, ASM detects the ASMLIB disks with a sector size = 4096 bytes (4kb) instead of a sector size = 512 bytes, this inconsistency generates the problem (ORA-15085: ASM disk "" has inconsistent sector size.):

 +ASM> select group_Number ,disk_number,path, sector_size from v$asm_disk
.
 GROUP_NUMBER DISK_NUMBER PATH                           SECTOR_SIZE
 ------------ ----------- ------------------------------ -----------
 ...
            0          75 ORCL:TMSTS01_LUN010                   4096

E) But if the ASMLIB is disabled/bypassed (ASM_DISKSTRING = '/dev/oracleasm/disks/*'), then the disks are presented/detected with a sector size = 512 bytes:

 +ASM> select group_Number ,disk_number,path, sector_size from v$asm_disk
.
 GROUP_NUMBER DISK_NUMBER PATH                                     SECTOR_SIZE
 ------------ ----------- ---------------------------------------- -----------
 ...
            0          75 /dev/oracleasm/disks/TMSTS01_LUN010                  512

D) This problem is due to the SAN Firware upgrade and due to the next ASMLIB bug (ASMLIB enhancement):

• Bug : 13999609 - PHYSICAL BLOCK SIZE REPORTED CAN CAUSE ISSUES WITH 10G DATABASES

• oracleasm driver (ASMLIB) as of today works with the expectation that logical block size and physical block size are 512/512 bytes. This particular storage/firmware appears doesn't do that, so the configuration won't work with oracleasm driver.

• We understand that 4K sector size disks need to be supported, so we are working on an ASMLIB enhancement fix.

• Therefore, at this moment this SAN firmware upgrade is not certified and is not supported with ASMLIB.

Symptoms

Error Description:

15085, 00000, "ASM disk \"%s\" has inconsistent sector size."
// *Cause:  An attempt to mount a diskgroup failed because a disk reported
//          inconsistent sector size value.
// *Action: Use disks with sector size consistent with Diskgroup sector size,
//          or make sure the operating system can accurately report the disk 
//          sector size.
//

Workaround

********* Warning !!!!!  ********* : Never set the "_disk_sector_size_override"=TRUE parameter in the ASM instance(s) as a workaround, since this parameter will corrupt the ASM diskgroups.

Therefore, the only valid workarounds are as follow:

1) Downgrade/rollback the SAN firmware upgrade.

2) Or disable/bypass the ASMLIB as follows:

ASM_DISKSTRING = '/dev/oracleasm/disks/*'

Patches

 3) Or apply the ASMLIB enhancement Patch: 13999609 as soon as it is released & enable back the ASMLIB API (as a permanent fix).

Note: ASMLIB development team confirmed that they are working on an ASMLIB enhancement (which will not be released soon due to it is not an easy fix and it will require extensive testing to support older SAN configurations as well) to support the changed introduced by the SAN firmware upgrade, for this reason they stated that at this moment this SAN firmware upgrade is not certified and is not supported with ASMLIB.

History

22-OCT-2012 : This document was created.