ZFS inifiniband traffic

When using a ZFS appliance with Exadata- Infiniband traffic is severely affected [ID 1351805.1]

Applies to:

Sun ZFS Storage 7320 - Version Not Applicable and later
Exadata Database Machine X2-2 Qtr Rack - Version Not Applicable and later
Oracle Exadata Storage Server Software - Version 11.2.2.3.0 and later
Information in this document applies to any platform.
ASM has issues in communicating with the cells and celldisks:
ORA-15080: synchronous I/O operation to a disk failed

At the server level- the NFS mounts on the ZFS Storage server also suffer.
nfs:nfs_wait_on_write_congestion

The entire Exadata cluster will either crash or be severely impaired

Symptoms

ASM has issues in communicating with the cells and celldisks:

ORA-15080: synchronous I/O operation to a disk failed

 

 

At the storage server level- the NFS mounts on the ZFS Storage server also suffer.

 

In /var/log/messages:

nfs:nfs_wait_on_write_congestion

 

 

The entire Exadata cluster will either crash or be severely impaired

IB related messages will be found in many locations.

Changes

If the ZFS Storage Appliance is running on software release <Q3.3 , then this issue can occur.

Cause

The Firmware/Storage software on the ZFS Storage Appliance can cause IB network collisions.

Solution

Upgrade the ZFS Storage Appliance to software version Q3.3 or later.