high CPU usage in AIX 7.1

Performance diminishes over time on AIX 7.1 due to Large Segment Aliasing [ID 1467807.1]

Applies to:

Oracle Server - Enterprise Edition - Version 10.2.0.1 to 11.2.0.3 [Release 10.2 to 11.2]
IBM AIX on POWER Systems (64-bit)

Symptoms

Database performance progressively degrades over time until the instance is restarted.
This could manifested by seeing the same SQL, with the same number of buffer get, s gradually  taking more and more CPU time, thereby increasing the overall elapsed time.  

This addition to CPU will show up under sys time and not user time as shown in the vmstat information bleow:

 

Changes

This problem may be seen after migration to AIX 7.1 from lower versions of the OS.
It will usually only be experienced in RDBMS 11.2.0.3 as a result of a code change that helped to  expose the issue.
RDBMS 11.2.0.2 and lower will not be likely to experience the same performance issue

Cause

This was raised as Bug 13877328 DATABASE PERFORMANCE SLOW DOWN STEADILY OVER THE TIME

In AIX7.1  Large Segment aliasing (terabyte aliasing) is  enabled by default.
In AIX 6, it is not enabled by default and therefore the problem is not expectd to be experienced
Generally this problem is more visible with large page sizes of 16M, but may also be experienced with smaller 4k pages.

Solution

 At time of writing , a fix is being worked on by IBM and should be released in AIX 7.1 TL1 SP5  in the later half of July, 2012.

A short term solution is to disable unshared aliases by doing the following:

vmo -r -o shm_1tb_unsh_enable=0

#Then do the following:
bosboot -a
sync;sync;sync
reboot -q