The binder was killed by a signal: Segmentation fault

Linking Fails With "ld: 0706-010 The binder was killed by a signal: Segmentation fault" On AIX 6.1 [ID 1264074.1]

Applies to:

Enterprise Manager Grid Control - Version: 10.2.0.2 to 10.2.0.5 - Release: 10.2 to 10.2
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.2.0.2   [Release: 10.2 to 11.2]
IBM AIX on POWER Systems (64-bit)
.
The problem below appears to affect Oracle installations on AIX servers running:

AIX 6.1 TL2 SP10 (6100-02-10-1036) and higher
AIX 6.1 TL3 SP7 (6100-03-07-1036) and higher
AIX 6.1 TL4 SP7 (6100-04-07-1036) and higher
AIX 6.1 TL5 SP3 (6100-05-03-1036) and higher
AIX 6.1 TL6 (all SP levels)
AIX 6.1 TL7 (all SP levels)

The problem does not appear to affect:

AIX 6.1 TL2 SP9 or lower
AIX 6.1 TL3 SP6 or lower
AIX 6.1 TL4 SP6 or lower
AIX 6.1 TL5 SP2 or lower

Symptoms

While installing Oracle10g Release 2 (10.2), Oracle11g Release 1 (11.1) or Oracle11g Release 2 (11.2) on an AIX server running AIX 6.1, the following error may occur:

INFO:
ld: 0706-010 The binder was killed by a signal: Segmentation faultCheck for binder messages or use local problem reporting procedures.

INFO: make: 1254-004 The error code from the last command is 254.

Stop.

INFO: make: 1254-004 The error code from the last command is 2.

Stop.

INFO: End output from spawned process.
INFO: ----------------------------------
INFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'agent nmb nmo nmhs' of makefile '/u01/app/oracle/product/11.2.0/sysman/lib/ins_emagent.mk'.

More specifically, the "ld: 0706-010" error may occur when running the following command:

% make -f $ORACLE_HOME/sysman/lib/ins_emagent.mk relink_sharedobj SHAREDOBJ=libnmemso

Using 'truss' to trace this command generates output similar to this:

:
15925416: 20512937: kopen("/usr/lib/glink64.o", O_RDONLY) = 31
15925416: 20512937: fstatx(31, 0x0FFFFFFFFFFFCBF0, 176, 0) = 0
15925416: 20512937: shmat(31, 0x0000000000000000, 6144) = 0x0700000300000000
15925416: 20512937: __libc_sbrk(0x0000000000010020) = 0x000000011161E140
15925416: 20512937: __libc_sbrk(0x0000000000010020) = 0x000000011162E160
15925416: 20512937: __libc_sbrk(0x0000000000010020) = 0x000000011163E180
15925416: 20512937: __libc_sbrk(0x0000000000010020) = 0x000000011164E1A0
15925416: 20512937: __libc_sbrk(0x0000000000010020) = 0x000000011165E1C0
15925416: 20512937: __libc_sbrk(0x0000000000010020) = 0x000000011166E1E0
15925416: 20512937: _getpid() = 15925416
15925416: 20512937: lseek(0x0000000000000003, 0x0000000000000000, 0x0000000000000001) = 0x0000000000002252
:
15925416: 20512937: kread(0x0000000000000003, 0x00000001102C14B8, 0x0000000000001000) = 0x0000000000001000
15925416: 0x00000000: "\n"
15925416: 20512937: shmat(4, 0x07F0000000000000, 2048) = 0x07F0000000000000
15925416: Received signal #11, SIGSEGV [default]
15925416: *** process killed ***
:

NOTE

To verify which AIX version is running, use the command:

% oslevel -s

Cause

Oracle Support opened Bug:10259393 for this issue but it appears to be a known issue on the following AIX 6.1 versions:

AIX 6.1 TL2 SP10 (6100-02-10-1036) and higher
AIX 6.1 TL3 SP7 (6100-03-07-1036) and higher
AIX 6.1 TL4 SP7 (6100-04-07-1036) and higher
AIX 6.1 TL5 SP3 (6100-05-03-1036) and higher
AIX 6.1 TL6 (all SP levels)
AIX 6.1 TL7 (all SP levels)

IBM have investigated the error and released fixes

Solution

For AIX 6.1 TL6, IBM have released fix IZ88711:

ftp://public.dhe.ibm.com/aix/efixes/iz88711/

Fixes for other AIX 6.1 TL's are also available.

Please contact IBM to identify the correct fix for your environment.


After installing the AIX fix, run the following command:

% make -f $ORACLE_HOME/sysman/lib/ins_emagent.mk relink_sharedobj SHAREDOBJ=libnmemso

to link the libnmemso.so library

Alternative solution
If these errors occur on AIX 6.1 TL4 SP6/SP7, you may alternatively be hitting the AIX defect below:

IZ44217 CRASH CAUSED BY STALE FILE DESCRIPTOR IN FORKED CHILD