java.lang.OutOfMemoryError: requested -4 bytes for size_t

Java SE Fails After Patch Installation with "java.lang.OutOfMemoryError: requested -4 bytes for size_t " or "Illegal Instruction" [ID 1276808.1]

Applies to:

Java Platform, Standard Edition - Version 1.4.2 to 1.5.0 [Release 1.4 to 1.5]
Oracle Solaris on x86 (32-bit)
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on SPARC (32-bit)

Symptoms

Java SE 1.4.2 and Java SE 5 fail after the Solaris 10 recommended patchset was installed. The problem is usually not observed in the global zone. One or more non-global zones are affected, not necessarily all non-global zones.

Error message from Java SE 5 (this example is from 1.5.0_21):

Exception java.lang.OutOfMemoryError: requested -4 bytes for size_t in
/BUILD_AREA/jdk1.5.0_21/hotspot/src/os/solaris/vm/os_solaris.cpp. Out of swap
space?

and/or (this example is from Java SE 1.5.0_22):

# ./java -Xms32m -Xmx32m -version
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# Internal Error (4F533F534F4C415249533F53504152430E4350500054 FF), pid=27154, tid=1
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode)
# An error report file with more information is saved as hs_err_pid27154.log
Illegal Instruction

Error message from Java SE 1.4.2:

Exception java.lang.OutOfMemoryError: requested -4 bytes for size_t in /export1/jdk142-update/ws/fcs/hotspot/src/os/solaris/vm/os_solaris.cpp. Out of swap space?

Error message from Java SE 5 pkgadd installation (shell script debugging output of Java SE 1.5.0_22):

+ jre1.5.0_22/bin/java com.sun.servicetag.Installer -source jre-1_5_0_22-fcs-bin-b03-solaris-sparc-09_oct_2009.bin

Illegal Instruction

Changes

This problem shows up after patch sets including the S10U8 rejuvenated kernel update patch were installed in the global zone, 141444-09 (Solaris 10 Sparc) or 141445-09 (Solaris 10 on x86) .

Cause

In case the problem shows up in a Solaris 8 or Solaris 9 branded zone: 

The most likely root cause is Solaris CR 6879229 Failures during execution of getpagesizes testcases in Solaris 9 BrandZ in s10u8.

The problem can also show up if patches that contain the fix for 6879229 were applied first, and then older branded zones packages were installed.

The correct version of the branded zones packages must be installed. Say, if a system is installed with Solaris 10 Update 10, then the SUNWs9brandr and SUNWs9brandu packages from Solaris 10 Update 10 must be installed, and to ensure a consistent patch state preferably before any patches are applied.

In case the problem shows up in a Solaris 10 whole root or sparse zone:

The most likely root cause is a failed patch installation; typically of 141444-09 or 141445-09 or patches required to install one of these patches.

• the patch state of the global zone is different to the patch state of the non-global zone.
• or patches were installed incompletely, resulting in a mismatch of libc and the kernel.

Other possible root causes

• libc.so.1 was replaced manually
• Environment variables were set which cause an incompatible libc.so.1 to be loaded

Solution

Temporary workaround:

• "java -XX:-UseLargePages" and "java -client" are not affected

In case the problem shows up in a Solaris 8 or Solaris 9 branded zone:

First, make sure that the correct version of the branded zones packages SUNWs9brandr and SUNWs9brandu were installed, and that if
patches were applied that modify these packages, they were applied after SUNWs9brandr and SUNWs9brandu were installed.

Then:

• On Sparc, install 143357-03 or higher into the global zone, and then reboot the global zone.
• On x86, install 142540-05 or higher into the global zone, and then reboot the global zone.
• The reboot of the global zone is absolutely necessary to activate the fix.

 

In case the problem shows up in a Solaris 10 whole root or sparse zone:

1. Check that the installed kernel patch revision (as per "showrev -p" and "ls -lrt /var/sadm/patch", NOT "uname -a") in the global zone is the same as in the non-global zone.
2. Check the verbose recommended patch cluster log file (in the /var/sadm/install_data directory) for error messages.
3. Check that libc libraries in the non-global zone are identical to the ones in the global zone.
4. Check that kernel patches, especially 141444-09 (Solaris 10 on Sparc) resp.141445-09 (Solaris 10 on x86) were installed completely.

Example how to check that libc libraries in the non-global zone are identical to the ones in the global zone. The example uses a whole root zone on Solaris 10 Update 8 on Sparc.

In the global zone:

checksum of 32-bit libc:
(global zone)% cksum /lib/libc.so.1
3795905205      1640076 /lib/libc.so.1

checksum of 64-bit libc:
(global zone)% cksum /lib/sparcv9/libc.so.1
1267745119      1779480 /lib/sparcv9/libc.so.1

Find mounted platform specific components of libc:
(global zone)% df -kl | grep libc
/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1 70589121 20009889 49873341    29%    /platform/sun4u-us3/lib/libc_psr.so.1
/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1 70589121 20009889 49873341    29%    /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1

Get checksums of those:
(global zone)% cksum /platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1
3524653149      6600    /platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1
(global zone)% cksum /platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
3966191396      6888    /platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1

Compare against the checksums in the non-global zone:
(non-global zone)% cksum /lib/libc.so.1
3795905205      1640076 /lib/libc.so.1
(non-global zone)% cksum /lib/sparcv9/libc.so.1
1267745119      1779480 /lib/sparcv9/libc.so.1
(non-global zone)% df -kl | grep libc
/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1 70589121 13643098 56240132    20%    /platform/sun4u-us3/lib/libc_psr.so.1
/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1 70589121 13643098 56240132    20%    /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
(non-global zone)% cksum /platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1
3524653149      6600    /platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap1.so.1
(non-global zone)% cksum /platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
3966191396      6888    /platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1

In this example, all checksums are identical.

The problem shows up, but the installed version of libc is correct

The problem can also be caused when LD_LIBRARY_PATH or other environment variables cause Java to load a non-default libc. To confirm that the correct version of libc.so.1 has been loaded, run:

% truss -edaf -o /tmp/truss.out java -version

On Solaris, libc.so defaults to the /usr/lib directory.