Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1514508 - [RFE] allow registrations to occur without blocking on task completion
Summary: [RFE] allow registrations to occur without blocking on task completion
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Registration
Version: 6.2.12
Hardware: Unspecified
OS: Unspecified
high
high vote
Target Milestone: 6.2
Assignee: Chris Duryee
QA Contact: jcallaha
URL:
Whiteboard:
: 1438945 1523254 (view as bug list)
Depends On:
Blocks: 1122832 1490019
TreeView+ depends on / blocked
 
Reported: 2017-11-17 15:52 UTC by Chris Duryee
Modified: 2018-12-06 20:45 UTC (History)
24 users (show)

Fixed In Version: rubygem-katello-3.0.0.168-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1553861 1553862 1553869 1553871 (view as bug list)
Environment:
Last Closed: 2018-05-21 20:16:43 UTC


Attachments (Terms of Use)
verification gif (deleted)
2018-04-27 19:04 UTC, jcallaha
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1672 None None None 2018-05-21 20:17:45 UTC
Foreman Issue Tracker 21703 None None None 2017-11-17 21:17:56 UTC
Foreman Issue Tracker 21797 None None None 2018-02-20 16:10:01 UTC

Description Chris Duryee 2017-11-17 15:52:42 UTC
Description of problem:

Currently, the system registration and unregistration processes create tasks, and then wait for those tasks to complete before returning to subscription-manager.

This usually works fine, but if the satellite is heavily loaded, registration requests may hold passenger workers for a long period of time while tasks are processed. This can eventually cause the satellite to run out of passenger workers and will tip over. Additionally, if http requests get backed up and take all queue slots, pulp API requests will become unresponsive. Many tasks (including system registration) require Pulp to accept API requests, so this can result in the satellite getting wound around the axle.

If registrations and unregistrations are done without blocking on tasks, this avoids this issue. It's technically not an issue if unregistrations occur via task, but if a register happens without a task, we would want the unregister to happen in the same time frame to ensure the subscription is freed before its consumed elsewhere.

Version-Release number of selected component (if applicable): 6.2.12


How reproducible: every time


Steps to Reproduce:

** note ** these steps describe how to make the satellite slow but not fall over from registrations, so the issue is more apparent. Once it falls over, there is not much to observe :)

1. set up a new satellite and configure it to have 40 passenger workers. The number isn't important as long as it's higher than the number in step 2
2. set up 15 clients to register to the Satellite in a loop, using subscrpition-mangaer register --force
3. on the satellite, run 'watch "passenger-status --show=requests | grep -e 'uri\|connected'" '. This will show the response times for the registration requests as they come in.

Actual results: lots of registration requests over 10 seconds, some over 20-30 seconds. Lots of running/pending tasks in the task list.


Expected results: registrations should get processed within a few seconds so as not to hold up passenger workers

Comment 3 Mike McCune 2017-12-11 16:34:06 UTC
*** Bug 1438945 has been marked as a duplicate of this bug. ***

Comment 10 pm-sat@redhat.com 2018-02-15 13:15:38 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/21703 has been resolved.

Comment 16 Brad Buckingham 2018-03-05 15:57:20 UTC
*** Bug 1523254 has been marked as a duplicate of this bug. ***

Comment 23 jcallaha 2018-04-27 19:04:24 UTC
Verified in Satellite 6.2.15 Snap 2

I loosely followed the steps outlined in the description.

I increased the passengers to 40
Then ran a loop of 150 hosts creating, registering, and destroying themselves in batches of 15.

See the attached gif for a recording of the grep'd passenger-status.

Comment 24 jcallaha 2018-04-27 19:04:59 UTC
Created attachment 1427796 [details]
verification gif

Comment 27 errata-xmlrpc 2018-05-21 20:16:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1672


Note You need to log in before you can comment on or make changes to this bug.