Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1516401

Summary: Refresh targets can grow unbounded causing Postgres InternalError
Product: Red Hat CloudForms Management Engine Reporter: Adam Grare <agrare>
Component: ProvidersAssignee: Adam Grare <agrare>
Status: CLOSED CURRENTRELEASE QA Contact: Dave Johnson <dajohnso>
Severity: urgent Docs Contact:
Priority: high    
Version: 5.9.0CC: cpelland, gblomqui, jfrey, jhardy, obarenbo
Target Milestone: GAKeywords: TestOnly
Target Release: 5.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 5.10.0.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1517962 (view as bug list) Environment:
Last Closed: 2018-06-21 20:41:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: Bug
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1517962    

Description Adam Grare 2017-11-22 14:40:13 UTC
Taking the uniq check out of EmsRefresh targets allows the number of targets in the args to grow unbounded to the point of running postgres out of memory for the transaction.

/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:598:in `async_exec': MIQ(MiqQueue.put_or_update) "PG::InternalError: ERROR:  invalid memory alloc request size 1073741824 (RuntimeError)
: UPDATE "miq_queue" SET "msg_data" = '\x04085b031e03375b0749221b4d616e61...

        from /opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction'
        from /opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/transactions.rb:211:in `transaction'
        from /opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/transactions.rb:392:in `with_transaction_returning_status'
        from /opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/persistence.rb:286:in `update!'
        from /var/www/miq/vmdb/app/models/miq_queue.rb:359:in `block in put_or_update'
        from /var/www/miq/vmdb/app/models/miq_queue.rb:337:in `loop'
        from /var/www/miq/vmdb/app/models/miq_queue.rb:337:in `put_or_update'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:173:in `queue_merge'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:60:in `block in queue_refresh'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:58:in `each'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:58:in `collect'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:58:in `queue_refresh'
        from /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresh_worker/runner.rb:20:in `block in do_before_work_loop'
        from /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresh_worker/runner.rb:17:in `each'
        from /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresh_worker/runner.rb:17:in `do_before_work_loop'
        from /var/www/miq/vmdb/app/models/miq_worker/runner.rb:148:in `prepare'
        from /var/www/miq/vmdb/app/models/miq_worker/runner.rb:126:in `start'

Comment 3 Adam Grare 2017-11-22 14:44:01 UTC
This is related to (and possibly a duplicate of) https://bugzilla.redhat.com/show_bug.cgi?id=1514595

It is unclear at this point if this is the root cause or a side-effect

Comment 5 CFME Bot 2017-11-27 17:16:35 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/d100b568ecd9d759e6239051e45658baadd65afe

commit d100b568ecd9d759e6239051e45658baadd65afe
Author:     Adam Grare <agrare@redhat.com>
AuthorDate: Thu Nov 9 08:50:09 2017 -0500
Commit:     Adam Grare <agrare@redhat.com>
CommitDate: Wed Nov 22 09:59:01 2017 -0500

    Uniq ems_refresh targets if over a threshold
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1516401

 app/models/ems_refresh.rb | 1 +
 1 file changed, 1 insertion(+)