Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1358963 - ec2 event monitor spams database connection errors
Summary: ec2 event monitor spams database connection errors
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.5.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: GA
: 5.5.5
Assignee: Joe Rafaniello
QA Contact: Matouš Mojžíš
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-21 23:33 UTC by Matouš Mojžíš
Modified: 2016-09-06 14:26 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-26 15:37:36 UTC
Category: ---
Cloudforms Team: ---


Attachments (Terms of Use)
evm.log (deleted)
2016-07-21 23:33 UTC, Matouš Mojžíš
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1488 normal SHIPPED_LIVE CFME 5.5.5 bug fixes and enhancement update 2016-07-26 19:28:27 UTC

Description Matouš Mojžíš 2016-07-21 23:33:41 UTC
Created attachment 1182672 [details]
evm.log

Description of problem:
I couldn't reproduce it in 5.5.4.2, so it's broken in 5.5.5.x only.
EventCatcher is not working properly, so I have to do manual refresh and automatic refresh/discovery doesn't work.
It doesn't spam it every five seconds, but it's like 1ten times per minute.

Version-Release number of selected component (if applicable):
5.5.5.0

How reproducible:
Always

Steps to Reproduce:
1. Add an ec2 provider
2.
3.

Actual results:
evm.log in attachment.



Expected results:


Additional info:

Comment 5 CFME Bot 2016-07-22 17:41:56 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a2e0ab9f1ae842ae6eac987b6c8b1c4b20cf63be

commit a2e0ab9f1ae842ae6eac987b6c8b1c4b20cf63be
Author:     Joe Rafaniello <jrafanie@redhat.com>
AuthorDate: Fri Jul 22 12:44:53 2016 -0400
Commit:     Joe Rafaniello <jrafanie@redhat.com>
CommitDate: Fri Jul 22 12:44:53 2016 -0400

    Accept an input vmdb config instead of always hitting the db
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963

 lib/vmdb/util.rb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 6 CFME Bot 2016-07-22 17:42:00 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=eff2db4d0ca449481db3db0950ae2ff4cd19dcce

commit eff2db4d0ca449481db3db0950ae2ff4cd19dcce
Author:     Joe Rafaniello <jrafanie@redhat.com>
AuthorDate: Fri Jul 22 12:36:59 2016 -0400
Commit:     Joe Rafaniello <jrafanie@redhat.com>
CommitDate: Fri Jul 22 12:36:59 2016 -0400

    Bump the connection pool size since 1 is excessive
    
    1 connection means you can't create a second thread and do any database query.
    It's just too easy to accidentally issue a query while not knowing this constraint.
    
    Note, this change would require you update the configuration for each appliance
    so, the next commit will prevent the db connection used in a thread.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963

 config/database.pg.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 7 CFME Bot 2016-07-22 17:42:04 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=7da8181a338c77f55ff160f09e8a496a74609fd0

commit 7da8181a338c77f55ff160f09e8a496a74609fd0
Author:     Joe Rafaniello <jrafanie@redhat.com>
AuthorDate: Fri Jul 22 12:49:29 2016 -0400
Commit:     Joe Rafaniello <jrafanie@redhat.com>
CommitDate: Fri Jul 22 13:00:37 2016 -0400

    Create the Stream with the http_proxy_uri using the existing @vmdb_config
    
    The @vmdb_config is created/updated in the MAIN thread of all
    MiqWorker::Runner during sync_config[1] and contains the http_proxy_uri we need.
    We already have this configuration from the main thread, so we can pass along
    the calculated http_proxy_uri to the Stream.
    
    This prevents the Thread created in
    ManageIQ::Providers::BaseManager::EventCatcher::Runner#start_event_monitor[2]
    from connecting to the database.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963
    
    [1] https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/blob/63646d7bf3472f2034eaf6b8510ced3afaddcd97/app/models/miq_worker/runner.rb#L284
    [2] https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/blob/77ef3f32c330d45414e9b3e3857ef1d2cb97fd73/app/models/manageiq/providers/base_manager/event_catcher/runner.rb#L114-128

 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb | 3 ++-
 .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb | 6 ++++--
 2 files changed, 6 insertions(+), 3 deletions(-)

Comment 8 CFME Bot 2016-07-22 17:42:09 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=c3cd7e2bade964fddd22acc5454ad5e683174f72

commit c3cd7e2bade964fddd22acc5454ad5e683174f72
Merge: c917133 7da8181
Author:     Gregg Tanzillo <gtanzill@redhat.com>
AuthorDate: Fri Jul 22 13:38:07 2016 -0400
Commit:     Gregg Tanzillo <gtanzill@redhat.com>
CommitDate: Fri Jul 22 13:38:07 2016 -0400

    Merge branch 'aws_event_catcher_dont_connect_to_db_in_thread' into '5.5.z'
    
    Don't connect to the database during the AWS event catcher subthread
    
    Note, this occurs only in 5.5.5+ via https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/commit/77ef3f32c330d45414e9b3e3857ef1d2cb97fd73
    
    A different solution will need to be done in upstream.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963
    
    
    * Bump the connection pool size since 1 is excessive  (this forces users/customers to update all appliance configurations, so follow up commits were done)
    
    * Update a method so you can accept an input vmdb config instead of always hitting the db to get one
    
    * Create the Stream with the http_proxy_uri using the existing @vmdb_config (avoids hitting the db in the subthread)
    
    Please review each commit.
    
    I tested this manually by creating a bogus aws cloud provider, changed verify_credentials? to always return true for aws, then restarting the appliance.  It fails at boot each time it tries to start the amazon event catcher worker.
    
    
    See merge request !1022

 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb | 3 ++-
 .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb | 6 ++++--
 config/database.pg.yml                                              | 2 +-
 lib/vmdb/util.rb                                                    | 4 ++--
 4 files changed, 9 insertions(+), 6 deletions(-)

Comment 10 Matouš Mojžíš 2016-07-25 13:01:46 UTC
Verified in: 5.5.5.4. After adding ec2 provider and waiting for 30 minutes no errors appeared in log.

Comment 12 errata-xmlrpc 2016-07-26 15:37:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1488.html

Comment 13 CFME Bot 2016-09-06 14:26:52 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=62e98284023e17d4466258857ede942983e6db69

commit 62e98284023e17d4466258857ede942983e6db69
Merge: 75afa46 ed6cc46
Author:     Greg Blomquist <gblomqui@redhat.com>
AuthorDate: Tue Sep 6 10:25:40 2016 -0400
Commit:     Greg Blomquist <gblomqui@redhat.com>
CommitDate: Tue Sep 6 10:25:40 2016 -0400

    Merge branch 'fix_aws_sns_topic_lost' into '5.5.z'
    
    fix wrong not working event catcher for aws
    
    before the default aws_config topic would be set to nil by a wrong call to Stream.new
    
    this was introduced in 7da8181a338c77f55ff160f09e8a496a74609fd0
    
    the original was a fix for
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963
    
    this one fixes
    https://bugzilla.redhat.com/show_bug.cgi?id=1367872
    
    See merge request !1041

 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb   | 2 +-
 .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

Comment 14 CFME Bot 2016-09-06 14:26:58 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=ed6cc46062232bc3545bb279a71c97cc773bd87f

commit ed6cc46062232bc3545bb279a71c97cc773bd87f
Author:     Marcel Hild <hild@b4mad.net>
AuthorDate: Wed Aug 17 19:09:22 2016 +0200
Commit:     Marcel Hild <hild@b4mad.net>
CommitDate: Wed Aug 17 19:09:22 2016 +0200

    before the default aws_config topic would be set to nil by a wrong
    call to Stream.new
    
    this was introduced 7da8181a338c77f55ff160f09e8a496a74609fd0
    
    the original was a fix for
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963
    
    this one fixes
    https://bugzilla.redhat.com/show_bug.cgi?id=1367872

 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb   | 2 +-
 .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)


Note You need to log in before you can comment on or make changes to this bug.