Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1535720 - Memory Leak in MiqServer process
Summary: Memory Leak in MiqServer process
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.9.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: GA
: 5.10.0
Assignee: Nick LaMuro
QA Contact: Dave Johnson
URL:
Whiteboard:
: 1511136 1511859 1511890 (view as bug list)
Depends On:
Blocks: 1536658 1536672 1536692
TreeView+ depends on / blocked
 
Reported: 2018-01-17 23:59 UTC by Nick LaMuro
Modified: 2018-10-16 16:12 UTC (History)
8 users (show)

Fixed In Version: 5.10.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1536658 1536672 1536692 (view as bug list)
Environment:
Last Closed: 2018-06-21 20:28:59 UTC
Category: ---
Cloudforms Team: CFME Core


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3618221 None None None 2018-10-16 16:12:22 UTC

Description Nick LaMuro 2018-01-17 23:59:08 UTC
Description of problem:

Memory Leak in the main "MIQ Server" process over a extended period of time.


Version-Release number of selected component (if applicable): 5.9.0


How reproducible:

Run any server over a large stretch of time, and the memory will slowly leak.  More workers on the same box will increase the rate at witch it leaks.


Steps to Reproduce:
1. Boot an appliance
2. Watch it leak (takes a few days)

If you wish to speed up this process, you can run the following from the /var/www/miq/vmdb directory:

    $ bin/rails runner "cfg = MiqServer.my_server.get_config; cfg.config[:server][:monitor_poll] = '1.second'; cfg.config[:server][:worker_monitor_frequency] = '1.second'; MiqServer.my_server.set_config cfg"



Actual results:

Server leaks around 10MB every 10 minutes or so, depending on the number of workers configured for that appliance.


Expected results:

No Leak.

Comment 2 Nick LaMuro 2018-01-18 00:04:43 UTC
We have a set of fixes already in place for this issue:

https://github.com/ManageIQ/manageiq/pull/16837
https://github.com/ManageIQ/manageiq-api/pull/288
https://github.com/ManageIQ/manageiq-automation_engine/pull/146
https://github.com/ManageIQ/manageiq-ui-classic/pull/3266


Turned out to be a small, unknown leak in ruby that was prevalent in our application since we defer loading a lot of libraries until they are needed using `require` inside of a method call, and that we were using `Pathnames` in our Rails `autoload_path` that was eventually getting added to the `$LOAD_PATH` in ruby.

This would make it so that every `require` call would leak a small amount of memory, and would never be released.  This only happens when Pathnames are used instead of raw strings in the `$LOAD_PATH`, so since both are valid, switching to using raw strings is the workaround we are using to prevent the leak.

Comment 6 dmetzger 2018-02-23 13:42:19 UTC
*** Bug 1511859 has been marked as a duplicate of this bug. ***

Comment 7 dmetzger 2018-02-23 13:44:28 UTC
*** Bug 1511890 has been marked as a duplicate of this bug. ***

Comment 8 Joe Rafaniello 2018-02-28 18:25:57 UTC
*** Bug 1511136 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.