Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1365149 - Multiple replication workers being spawned in stopping state that never go away
Summary: Multiple replication workers being spawned in stopping state that never go away
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Replication
Version: 5.4.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.7.0
Assignee: Gregg Tanzillo
QA Contact: Alex Newman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-08 13:30 UTC by Jared Deubel
Modified: 2017-12-05 05:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-06 20:29:28 UTC
Category: ---
Cloudforms Team: ---


Attachments (Terms of Use)
Hot fix for replication worker management (deleted)
2016-08-08 14:34 UTC, Gregg Tanzillo
no flags Details
Rubyrep nested transaction fix (deleted)
2016-08-08 14:43 UTC, Gregg Tanzillo
no flags Details | Diff
Hot fix for replication worker management (deleted)
2016-08-08 21:03 UTC, Gregg Tanzillo
no flags Details

Description Jared Deubel 2016-08-08 13:30:36 UTC
Description of problem:

This has created 800 connections to the db and then stops replication from working: 

Snippet showing replicate processes:

root@test_cfme_db_rep:/root # ps -ef | grep replicat
root      3869  3819  1 05:24 ?        00:08:59 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      4298  3819  0 05:31 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      4602  3819  0 05:37 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      4927  3819  0 05:44 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      5156  3819  0 05:49 ?        00:00:35 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      5573  3819  0 05:59 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      6041  3819  0 06:08 ?        00:00:12 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      6350  3819  0 06:14 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      6595  3819  0 06:20 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      7113  3819  0 06:31 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      7363  3819  0 06:37 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      8082  3819  0 06:52 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      8386  3819  0 06:58 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      8692  3819  0 07:04 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      9033  3819  0 07:11 ?        00:00:16 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root      9495  3819  0 07:22 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     10097  3819  0 07:35 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     10548  3819  0 07:46 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     10884  3819  0 07:52 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     11177  3819  0 07:59 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     11460  3819  0 08:05 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     11725  3819  0 08:10 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     12920  3819  0 08:36 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     13657  3819  0 08:53 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     13918  3819  0 08:58 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     14371  3819  0 09:06 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     14846  3819  0 09:16 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     15142  3819  0 09:23 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     15396  3819  0 09:29 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     15648  3819  0 09:34 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     15947  3819  0 09:41 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     16853  3819  0 10:00 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     17190  3819  0 10:07 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     17680  3819  0 10:17 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     17931  3819  0 10:23 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     18937  3819  0 10:44 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     19396  3819  0 10:54 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     19644  3819  0 11:00 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     20081  3819  0 11:08 ?        00:00:14 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     20560  3819  0 11:18 ?        00:00:15 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate
root     21514  3819  0 11:38 ?        00:00:13 ruby /opt/rh/cfme-gemset/bin/rake evm:dbsync:replicate




Version-Release number of selected component (if applicable):
5.4

Comment 5 Gregg Tanzillo 2016-08-08 14:34:22 UTC
Created attachment 1188803 [details]
Hot fix for replication worker management

Attached is a hot fix for fixing the issue with replication worker process management. Please follow these instructions to apply:

1. Copy bz-1365149-fix-repl-worker-shutdown.tar to /var/www/miq/vmdb on the appliance.
2. Create backups of the following 2 file (new versions are included in the tar file)
app/models/miq_server/worker_management/monitor/stop.rb
lib/workers/replication_worker.rb
3. Unpack the tar file from the /var/www/miq/vmdb directory:
tar -xvf bz-1365149-fix-repl-worker-shutdown.tar

Comment 6 Gregg Tanzillo 2016-08-08 14:43:12 UTC
Created attachment 1188804 [details]
Rubyrep nested transaction fix

This fixes an issue where replication fails when in error is encountered in a nested transaction. Apply the patch with the following instructions:

1. Copy the patch to the regional appliance that is failing. You can just put it in the home directory like this:
scp transaction_fix.patch root@<hostname of regional appliance>:~

2. CD to the gem directory and apply the patch:
cd /opt/rh/cfme-gemset/bundler/gems/rubyrep-3dcf0d958aa1 && patch -p1 -b < ~/transaction_fix.patch

Comment 7 Gregg Tanzillo 2016-08-08 21:03:34 UTC
Created attachment 1188914 [details]
Hot fix for replication worker management

Attached is a hot fix for fixing the issue with replication worker process management. Please follow these instructions to apply:

1. Copy bz-1365149-fix-repl-worker-shutdown.tar to /var/www/miq/vmdb on the appliance.
2. Create backups of the following 2 file (new versions are included in the tar file)
app/models/miq_server/worker_management/monitor/stop.rb
lib/workers/replication_worker.rb
3. Unpack the tar file from the /var/www/miq/vmdb directory:
tar -xvf bz-1365149-fix-repl-worker-shutdown.tar


Note You need to log in before you can comment on or make changes to this bug.