Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1067260 - [GSS] (6.2.x) Thread leak and OutOfMemoryError on Tomcat using jboss-client.jar, calling an EJB on EAP
Summary: [GSS] (6.2.x) Thread leak and OutOfMemoryError on Tomcat using jboss-client.j...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Remoting
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: CR2
: EAP 6.2.2
Assignee: David M. Lloyd
QA Contact: Jitka Kozana
Russell Dickenson
URL:
Whiteboard:
Depends On:
Blocks: 1103785 1067264 1068785
TreeView+ depends on / blocked
 
Reported: 2014-02-20 03:22 UTC by Osamu Nagano
Modified: 2018-12-05 17:21 UTC (History)
5 users (show)

Doc Type: Bug Fix
Doc Text:
Fixes a memory leak that can result from pending connections in org.jboss.remoting3.remote.RemoteConnectionProvider
Clone Of:
: 1069075 (view as bug list)
Environment:
Last Closed: 2014-06-02 12:49:40 UTC
Type: Bug


Attachments (Terms of Use)
client side thread dump from the customer (deleted)
2014-02-20 03:22 UTC, Osamu Nagano
no flags Details
remoting 3.2.19 and jboss-client.jar for eap620 (deleted)
2014-02-21 00:27 UTC, Osamu Nagano
no flags Details

Description Osamu Nagano 2014-02-20 03:22:38 UTC
Created attachment 865334 [details]
client side thread dump from the customer

Description of problem:
There is a web app on Tomcat calling an EJB on EAP 6.2.0.  After a stress test there are hundreds of "ejb-client-context-tasks-NNN-thread-2" threads on Tomcat and results in OOME.

~~~
"ejb-client-context-tasks-463-thread-2" daemon prio=10 tid=0x00007fb800233000 nid=0x646c waiting on condition [0x00007fb7b888a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000e2a25810> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
   Locked ownable synchronizers:
	- None
~~~


Version-Release number of selected component (if applicable):
EAP 6.2.0 (JBoss Remoting 3.2.18.GA)
EAP 6.1.1 (JBoss Remoting 3.2.16.GA)


How reproducible:
We tried to reproduce but we couldn't.  Though it is always reproducible in the customer's env.  The attachement is a thread dump from the customer.  There are 37 samples and started to increase from 21st.  I made this internal because it includes some specific class names.


Additinal Info:
We dcided to debug in the customer's env.  Will follow up soon about it.

Comment 1 Osamu Nagano 2014-02-20 03:32:11 UTC
We got a heap dump and found that org.jboss.remoting3.remote.RemoteConnectionProvider.pendingInboundConnections is holding over 23 thausands IoFutures.  I found the following commit in the upstream, without JIRA, on the same class.

[Stop memory leak from pending connections]
https://github.com/jboss-remoting/jboss-remoting/commit/7d20e8ebc141cabafd5f13ed92b5703bc5579c39

I send a debug patch of JBoss Remoting 3.2.19.GA, which include the above commit, and corresponding jboss-client.jar.  The issue resolved, though not sure about the detailed logic.

Comment 2 Osamu Nagano 2014-02-20 03:36:00 UTC
% git log 3.2.18.GA..3.2.19.GA
commit 7bf6c480d7c640075c4692843767e5c355e0b2c4
Author: David M. Lloyd <david.lloyd@redhat.com>
Date:   Fri Dec 20 16:09:58 2013 -0600

    Prep 3.2.19.GA

commit 2665edffb42ca3add9222ed90721d4024a6a3584
Author: David M. Lloyd <david.lloyd@redhat.com>
Date:   Thu Nov 14 12:14:34 2013 -0600

    [REM3-176] Fix CME if some connections are being opened

commit 7d20e8ebc141cabafd5f13ed92b5703bc5579c39
Author: David M. Lloyd <david.lloyd@redhat.com>
Date:   Thu Nov 7 21:25:52 2013 -0600

    Stop memory leak from pending connections

commit 1fc3e59a31b84d69a3792a99bfd77f72c6f1accf
Author: David M. Lloyd <david.lloyd@redhat.com>
Date:   Tue Oct 29 17:11:57 2013 -0500

    Fix wrong name for AUTHORIZE_ID simple option

commit cec41089037f187c594cfc1e4501ba795518d4e7
Author: David M. Lloyd <david.lloyd@redhat.com>
Date:   Fri Oct 25 12:41:37 2013 -0500

    Next is 3.2.19.GA

Comment 4 Osamu Nagano 2014-02-21 00:27:53 UTC
Created attachment 865755 [details]
remoting 3.2.19 and jboss-client.jar for eap620

This is just a debug patch for EAP 6.2.0 to see the commit actually fixes or not.

Comment 5 Richard Janík 2014-03-05 18:19:55 UTC
I couldn't properly reproduce it in time, but since the issue was resolved I'll set this to verified.

Comment 6 Scott Mumford 2014-03-10 01:05:39 UTC
If this is an issue that requires a release note (as the requires_doc_text suggests) is it possible to get a few details added to the Doc Text field above to enable ECS to draft a release note?


Note You need to log in before you can comment on or make changes to this bug.