glassfish error message Interrupting idle Thread

GlassFish Server 3.x Log (server.log) Shows "Interrupting idle Thread" Messages [ID 1484459.1]

Applies to:

Oracle GlassFish Server - Version 3.1.2 and later
Information in this document applies to any platform.

Symptoms

The GlassFish Server 3.1.x server.log shows the following repeating entries as shown below:

[#|2012-05-18T09:34:02.686+0900|WARNING|oracle-glassfish3.1.2|com.sun.grizzly.
config.GrizzlyServiceListener|_ThreadID=24;_ThreadName=Thread-2;|GRIZZLY0023:
Interrupting idle Thread: http-thread-pool-28080(3).|#]
.
...
  .
[#|2012-05-18T10:10:11.093+0900|WARNING|oracle-glassfish3.1.2|com.sun.grizzly.
config.GrizzlyServiceListener|_ThreadID=24;_ThreadName=Thread-2;|GRIZZLY0023:
.

 

Cause

The GlassFish (GF) server log shows that some threads are being interrupted may be be cause by Bug 14228839 "INCORRECT TIMEOUT SWITCH LOGIC MAY CAUSE UNEXPECTED INTERRUPTION OF A THREAD".

In GF 3.1.x, the GlassFish HTTP component (Grizzly) has a request timeout mechanism that will timeout requests that are idle (not generating response). This timeout is set to 900 seconds (or 15 minutes) by default. For requests that are idle, they will be issued a thread interrupt by Grizzly to try atttempt to bail out from this situation. This behaviour does not exist in GF 2.x. So in GlassFish 3.x, a HTTP request thread can be interrupted by the Grizzly housekeeping.

The Bug 14228839 or GLASSFISH-18705 shows that the housekeeping of tracking the idle time is broken and is fixed in 3.1.2 patch 2. However, the version used is already GlassFish 3.1.2 Patch 2 or later and these error messages are happening, then it may mean that the HTTP request is somehow idle due to inactivity on this thread. For example, it is possible that this request calls a remote service that takes more than 15 minutes to respond. In this case, one may see these errors if the timeout is too small for this application.

For example, to set the the "server-config" such that not have any request timeout interruption, the following asadmin command can be use to set it (to unlimited).

# asadmin set server-config.network-config.protocols.protocol.<listener-name>.http.request-timeout-seconds=-1
   - NOTE: <listener-name> is the name of the listener they wish to disable the  timeout on.
   - (-1) means unlimited

NOTE: If there is indeed an application on a request thread that is idle/blocked and holding up the request thread, it is possible that interrupting this thread will cause this thread to go into a high CPU spin if this application is stalling due to waiting for a CORBA/IIOP response. This is related to Bug 14247062. The logging of this message with the appearance of a persistenting high CPU consumption on some threads may indicate this issue. A thread dump will confirm this.

Solution

1. Update to at least GlassFish Server 3.1.2 Patch 2 or later to avoid Bug 14228839.

2. If the version is GF 3.1.2 patch 2 or later but this is seen, then it means that this thread may have some have inactivity longer than the default 15 mins. You may want to
 

• Check this request is taking so long and see if this is a application issue. 
• Check if the property "http.request-timeout-seconds" needs to be set to raise the default request timeout of 900 seconds for the particular HTTP listener involved.