Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1512073 - java.lang.OutOfMemoryError: GC overhead limit exceeded
Summary: java.lang.OutOfMemoryError: GC overhead limit exceeded
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 12.0 (Pike)
Assignee: Michael Vorburger
QA Contact: Itzik Brown
URL:
Whiteboard:
: 1451401 1512074 (view as bug list)
Depends On:
Blocks: 1439320
TreeView+ depends on / blocked
 
Reported: 2017-11-10 19:00 UTC by Sai Sindhur Malleni
Modified: 2018-10-18 08:09 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
N/A
Last Closed: 2018-02-19 13:20:20 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenDaylight Bug NETVIRT-985 None None None 2017-11-27 14:36:15 UTC
OpenDaylight Bug OVSDB-435 None None None 2017-11-30 12:49:04 UTC

Description Sai Sindhur Malleni 2017-11-10 19:00:23 UTC
Description of problem:
On running perf/scale tests on an OSP 12 cloud with 3 OpenStack controllers + 3 ODLs + 28 computes, by creating 100s of neutron resources and deleting them we see OOM on an ODL instance.  

Version-Release number of selected component (if applicable):
OSP12
Puddle: 2017-10-31.2
ODL RPM: opendaylight-6.2.0-3.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. Run perf/scale tests using Browbeat+neutron and create large number of neutro nresources
2.
3.

Actual results:
one of the ODLs dies due to OOM

Expected results:
There should be no OOM

Additional info:

Comment 1 Sai Sindhur Malleni 2017-11-10 19:02:56 UTC
STDOUT of JVM when exiting: https://gist.github.com/smalleni/538bf2760ba9fbad5f47e76421fa6589

Comment 2 Sai Sindhur Malleni 2017-11-12 23:23:10 UTC
We hit this gain and here is a more complete output of the JVM before exiting
https://gist.github.com/smalleni/3b2febfca36c1a6ae5b41b295a9ebf84

Comment 3 Michael Vorburger 2017-11-27 14:35:51 UTC
Over the last 2 weeks, we've invested significant effort into addressing what we suspect was the root cause of this OOM (plugging MD SAL Transaction leaks found by using http://blog2.vorburger.ch/2017/09/how-to-find-transaction-related-memory.html) upstream under https://jira.opendaylight.org/browse/NETVIRT-985, and are currently awaiting confirmation from Reporter re. whether that did the trick and fixes this problem...

Comment 5 Michael Vorburger 2017-11-27 14:58:43 UTC
*** Bug 1512074 has been marked as a duplicate of this bug. ***

Comment 6 Mike Kolesnik 2018-02-19 12:38:52 UTC
Michael, has this been solved?

If so please update the bug appropriately

Comment 7 Mike Kolesnik 2018-02-19 12:39:00 UTC
*** Bug 1451401 has been marked as a duplicate of this bug. ***

Comment 8 Michael Vorburger 2018-02-19 13:14:49 UTC
Removing needinfo on my and putting back needinfo from smalleni .. will also email.

Comment 9 Sridhar Gaddam 2018-02-19 13:20:20 UTC
(In reply to Michael Vorburger from comment #3)
> Over the last 2 weeks, we've invested significant effort into addressing
> what we suspect was the root cause of this OOM (plugging MD SAL Transaction
> leaks found by using
> http://blog2.vorburger.ch/2017/09/how-to-find-transaction-related-memory.
> html) upstream under https://jira.opendaylight.org/browse/NETVIRT-985, and
> are currently awaiting confirmation from Reporter re. whether that did the
> trick and fixes this problem...

Closing this bug as there were many fixes done in Netvirt/OVSDB to handle these issues. We can re-open this or create a new one if we encounter the issue in future.


Note You need to log in before you can comment on or make changes to this bug.