ORA-481 ORA-29701

ORA-481 ORA-29701: LMON Crashes Instance After EAGAIN in sgipcnDSRecv [ID 1461897.1]

Applies to:

Oracle Server - Enterprise Edition - Version 11.2.0.2 and later
Information in this document applies to any platform.

Symptoms

11gR2 RAC instance crashes:

• alert_<ORACLE_SID>.log

Mon May 21 11:40:04 2012
Error 29701: unexpected return code 6 from the Cluster Synchronization Service
Errors in file /oratrace/racdb/dump/diag/rdbms/racdb/racdb3/trace/racdb3_lmon_11686.trc:
ORA-29701: unable to connect to Cluster Synchronization Service
Mon May 21 11:40:04 2012
USER (ospid: 4188): terminating the instance due to error 481
Mon May 21 11:40:04 2012
opiodr aborting process unknown ospid (15172) as a result of ORA-1092
Mon May 21 11:40:04 2012
ORA-1092 : opitsk aborting process
..
System State dumped to trace file /oratrace/racdb/dump/diag/rdbms/racdb/racdb3/trace/racdb3_diag_11676.trc
Mon May 21 11:40:15 2012
Termination issued to instance processes. Waiting for the processes to exit
..
License high water mark = 6778
Instance terminated by USER, pid = 4188
USER (ospid: 8140): terminating the instance

• <ORACLE_SID>_lmon_<pid>.trc

2012-05-21 11:40:04.488: [ GIPCNET] gipcmodNetworkProcessRecv: [network]  failed recv attempt endp 600000000031ccd0 [0000000000000018] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=3bde1c17-79b989d0-11686))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_nngmpdb3_)(GIPCID=79b989d0-3bde1c17-8960))', numPend 3, numReady 2, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 8960, flags 0x24a716, usrFlags 0x34000 }, req 60000000003131b0 [000000000007caa8] { gipcReceiveRequest : peerName '', data 60000000003464e8, len 10240, olen 0, off 0, parentEndp 600000000031ccd0, ret gipcretFail (1), objFlags 0x0, reqFlags 0x2 }*** 2012-05-21 11:40:04.553
2012-05-21 11:40:04.553: [ GIPCNET] gipcmodNetworkProcessRecv: slos op  :  sgipcnDSRecv
2012-05-21 11:40:04.553: [ GIPCNET] gipcmodNetworkProcessRecv: slos dep :  Resource temporarily unavailable (11)
2012-05-21 11:40:04.553: [ GIPCNET] gipcmodNetworkProcessRecv: slos loc :  recv
2012-05-21 11:40:04.553: [ GIPCNET] gipcmodNetworkProcessRecv: slos info:  failed
2012-05-21 11:40:04.564: [GIPCXCPT] gipcmodMuxCallbackRecv: internal receive request failed req 6000000000312d00 [000000000007cac3] { gipcReceiveRequest : peerName '', data 0000000000000000, len 0, olen 0, off 0, parentEndp 600000000031ccd0, ret gipcretFail (1), objFlags 0x0, reqFlags 0x4 }, ret gipcretFail (1)
2012-05-21 11:40:04.565: [ GIPCMUX] gipcmodMuxCallbackRecv: EXCEPTION[ ret gipcretFail (1) ]  error during recv on endp 600000000031ceb0
2012-05-21 11:40:04.565: [GIPCXCPT] gipcInternalSend: connection not valid for send operation endp 600000000031d270 [0000000000000087] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=4407aad8-79b989d0-11686))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_nngmpdb3_)(GIPCID=79b989d0-4407aad8-8960))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 8960, flags 0x3861e, usrFlags 0x20010 }, ret gipcretConnectionLost (12)
2012-05-21 11:40:04.565: [GIPCXCPT] gipcSendSyncF [clsssServerRPC : clsss.c : 6272]: EXCEPTION[ ret gipcretConnectionLost (12) ]  failed to send on endp 600000000031d270 [0000000000000087] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=4407aad8-79b989d0-11686))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_nngmpdb3_)(GIPCID=79b989d0-4407aad8-8960))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 8960, flags 0x3861e, usrFlags 0x20010 }, addr 0000000000000000, buf 9fffffffffffbe40, len 72, flags 0x8000000
2012-05-21 11:40:04.565: [ CSSCLNT]clsssServerRPC: send failed with err 12, msg type 10
2012-05-21 11:40:04.565: [ CSSCLNT]clssgsMbrPrivateInfo: RPC failure, rc 3
kgxgnprdata: error: status 3 (0 )
kjfmPriCheck: query for instance 4's private data failed
2012-05-21 11:40:04.579: [ CSSCLNT]clsssRecvMsg: got a disconnect from the server while waiting for message type 1
2012-05-21 11:40:04.579: [ CSSCLNT]clssgsGroupGetStatus:  communications failed (0/3/-1)
2012-05-21 11:40:04.579: [ CSSCLNT]clssgsGroupGetStatus: returning 8
kgxgnpstat: received ABORT event from CLSS
kjxgmpoll: kgxgnpstat returns ABORT
kjxgmpoll: kgxgnpstat return 6
LMON caught an error 29701 in the main loop
error 29701 detected in background process
ORA-29701: unable to connect to Cluster Synchronization Service

Cause

 bug 14096821

Solution

bug 14096821 is fixed in 11.2.0.4, at the time of this writing, please check and request patch 14096821 if it does not exist for your platform/version.

References

BUG:14096821 - LMON PROCESS DIED WITH ORA-29701