listener crashes

http://msutic.blogspot.com/2011/06/oracle-listener-crashes-with-core-dump.html

TNS Listener Crash with Core dump [ID 549932.1]

Applies to:

Oracle Net Services - Version: 10.2.0.3.0 to 11.1.0.6.0
Generic UNIX

Symptoms

• There may be heavy load on the CPU shooting up to 100%.
• The number of sessions in the database is well below the upper or maximum limit defined in the parameter  file.
• The listener crashes suddenly during  this heavy CPU load generating the core.
• (Optional) Listener.Ora has SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF.

Listener Log:
-------------

.....
.....
19-NOV-2007 13:40:49 * (CONNECT_DATA=(SID=ORAC)(CID=(PROGRAM=C:\pegasos\te\usys\bin\uniface.exe)(HOST=TERVI-NB179)(USER=kjokioja))) * (ADDRESS= (PROTOCOL=tcp)(HOST=10.12.152.5)(PORT=1670)) * establish * ORAC * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12571: TNS:packet writer failure
TNS-12560: TNS:protocol adapter error
TNS-00530: Protocol adapter error
Linux Error: 104: Connection reset by peer
19-NOV-2007 13:40:49 * (CONNECT_DATA=(SID=ORAC)(CID=(PROGRAM=C:\pegaos\te\usys\bin\uniface.exe)(HOST=TERVI-0184A)(USER=paitasal))) * (ADDRESS=(PROTO COL=tcp)(HOST=10.12.176.136)(PORT=1574)) * establish * ORAC * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe
.........
........                                                                                                            

The Operating system log (/var/log/messages) may show the following :

tnslsnr[5841]: segfault at 0000000000000018 rip 0000003eab66854d rsp 0000007fbfff9230 error 4
tnslsnr[6469]: segfault at 0000000000000018 rip 0000003eab66854d rsp 0000007fbfff9420 error 4
tnslsnr[7375]: segfault at 0000000000000018 rip 0000003eab668bb3 rsp 0000007fbfff9c70 error 4

Gdb on the core file generated shows the following stack :

#gdb  /home/oracle/db_1/bi/tnslsnr core.7375   Core was generated  by `/opt/oracle/ora102/bin/tnslsnr LISTENER -inherit'. 
    Program terminated with  signal 11, Segmentation fault. 
    Reading symbols from /opt/oracle/ora102/lib/libclntsh.so.10.1...(no debugging symbols found)...done.
    ........
    .........
        (gdb) bt
    #0  0x00000032b7468bb3 in _int_free () from  /lib64/tls/libc.so.6
    #1  0x00000032b74691f6 in free () from /lib64/tls/libc.so.6
    #2  0x000000000040f01f in nsglhe ()
    #3  0x00000000004114e9 in nsglma ()
    #4  0x00000000004061cb in main ()
    (gdb) where
    #0  0x00000032b7468bb3 in _int_free () from /lib64/tls/libc.so.6
    #1  0x00000032b74691f6 in free () from /lib64/tls/libc.so.6
    #2  0x000000000040f01f  in nsglhe ()
    #3  0x00000000004114e9 in nsglma ()
    #4  0x00000000004061cb in main ()
                                 

The core indicates that the program terminated with signal 11, Segmentation fault .
SIGSEGV is reported for improper memory handling .The default action for a program upon receiving
SIGSEGV is abnormal termination. This action will end the process.
The virtual memory stats (vmstat) output may show huge paging/swapping activity .

For Example:
-------------

$vmstat 5 7 The si and so column under the swap section suggests the amount of memory swapped in from disk (/s)and amount of memory swapped to disk (/s) respectively.

    procs                      memory    swap          io     system         cpu 
  r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id 
  2  0  0  27124   3940   1160  74528   0 1304  956  5360  272   191  51  21  98 
  3  0  0  27080   3756   1180  72580   0 260  2552   388  218   429  90  10  89 
 14  2  1  26808   5096   1188  69868  84 1108 2016  9064  490   567  59  22  96 
  4  0  0  25548   5912   1192  73032   0   0   436     0  478   736  50  50   0 
  5  0  0  25548   3940   1192  73548   0   0  1560     0  301   385  93   7   0 
  1  1  1  25548   3336   1192  71800   8 176  2848   432  258   147  91   9  88 
  1  0  0  25544   4124   1200  70480 116  60   836    60  171   200  97   3  92 

Note: You may also use the top command to check the system memory usage.

Cause

Extensive paging/swapping activity is a clear  indication that the sytem is running out of the physical memory.

Solution

Solution :

1. Increase the physical memory of the system.

                             OR

2. Apply the Patch 6139856 for unpublished Bug 6139856 if available for your platform.
                             OR
3. Configure Hugepages on the OS. Ref : Note 361323.1