Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1361007 - failed to stop Openstack instance
Summary: failed to stop Openstack instance
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Community
Component: general
Version: develop
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: 24.0
Assignee: matt jia
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-28 06:48 UTC by matt jia
Modified: 2017-02-21 18:48 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-21 18:48:11 UTC


Attachments (Terms of Use)

Description matt jia 2016-07-28 06:48:34 UTC
After an instance is launched, beakerd will try to stop it first. However, sometimes it is failed to stop because the OpenStack server is too slow. In this case, the instance will be deleted in the OpenStack which is not expected.

Comment 1 matt jia 2016-07-28 06:52:16 UTC
Some logs from the beakerd.log in my beaker testing env.

Jul 28 16:05:07 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:07 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1358
Jul 28 16:05:07 beaker beakerd[19527]: bkr.server.dynamic_virt INFO Created <Server: beaker-recipe-11>
Jul 28 16:05:12 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still building
Jul 28 16:05:12 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:13 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1398
Jul 28 16:05:18 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still building
Jul 28 16:05:18 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:19 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1556
Jul 28 16:05:19 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:19 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "POST /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a/action HTTP/1.1" 202 0
Jul 28 16:05:20 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:20 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:21 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:22 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:22 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:22 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:23 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:23 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:24 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:25 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:25 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:26 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:27 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:27 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:27 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:28 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:28 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:29 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:30 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:30 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:30 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:31 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:31 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:32 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:33 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:33 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:34 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:35 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:35 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:35 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:36 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:36 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:37 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
                                                                               Jul 28 16:05:51 beaker beakerd[19527]: bkr.server.dynamic_virt DEBUG <Server: beaker-recipe-11> still stopping
Jul 28 16:05:51 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:52 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "GET /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 200 1566
Jul 28 16:05:52 beaker beakerd[19527]: requests.packages.urllib3.connectionpool INFO Starting new HTTP connection (1): 10.8.188.11
Jul 28 16:05:52 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "DELETE /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 204 0
Jul 28 16:05:52 beaker beakerd[19527]: bkr.server.tools.beakerd ERROR Error in provision_virt_recipe(11)
Jul 28 16:05:52 beaker beakerd[19527]:  Traceback (most recent call last):
Jul 28 16:05:52 beaker beakerd[19527]:    File "/usr/lib/python2.6/site-packages/bkr/server/tools/beakerd.py", line 601, in provision_virt_recipes
Jul 28 16:05:52 beaker beakerd[19527]:      provision_virt_recipe(recipe_id)
Jul 28 16:05:52 beaker beakerd[19527]:    File "/usr/lib/python2.6/site-packages/bkr/server/tools/beakerd.py", line 638, in provision_virt_recipe
Jul 28 16:05:52 beaker beakerd[19527]:      instance_id = manager.create_vm(vm_name, flavor)
Jul 28 16:05:52 beaker beakerd[19527]:    File "/usr/lib/python2.6/site-packages/bkr/server/dynamic_virt.py", line 56, in create_vm
Jul 28 16:05:52 beaker beakerd[19527]:      self._wait_for_stop(instance)
Jul 28 16:05:52 beaker beakerd[19527]:    File "/usr/lib/python2.6/site-packages/bkr/server/dynamic_virt.py", line 88, in _wait_for_stop
Jul 28 16:05:52 beaker beakerd[19527]:      % (instance, instance.status))

Comment 2 matt jia 2016-07-28 06:52:59 UTC
Jul 28 16:05:52 beaker beakerd[19527]: requests.packages.urllib3.connectionpool DEBUG "DELETE /v2/6385e9ac335a404c8220aca7c3f44b72/servers/cae637e3-8aed-42be-83ec-76531ca62a6a HTTP/1.1" 204 0
Jul 28 16:05:52 beaker beakerd[19527]: bkr.server.tools.beakerd ERROR Error in provision_virt_recipe(11)
Jul 28 16:05:52 beaker beakerd[19527]:  Traceback (most recent call last):
Jul 28 16:05:52 beaker beakerd[19527]:    File "/usr/lib/python2.6/site-packages/bkr/server/tools/beakerd.py", line 601, in provision_virt_recipes
Jul 28 16:05:52 beaker beakerd[19527]:      provision_virt_recipe(recipe_id)
Jul 28 16:05:52 beaker beakerd[19527]:    File "/usr/lib/python2.6/site-packages/bkr/server/tools/beakerd.py", line 638, in provision_virt_recipe
Jul 28 16:05:52 beaker beakerd[19527]:      instance_id = manager.create_vm(vm_name, flavor)
Jul 28 16:05:52 beaker beakerd[19527]:    File "/usr/lib/python2.6/site-packages/bkr/server/dynamic_virt.py", line 56, in create_vm
Jul 28 16:05:52 beaker beakerd[19527]:      self._wait_for_stop(instance)
Jul 28 16:05:52 beaker beakerd[19527]:    File "/usr/lib/python2.6/site-packages/bkr/server/dynamic_virt.py", line 88, in _wait_for_stop
Jul 28 16:05:52 beaker beakerd[19527]:      % (instance, instance.status))
Jul 28 16:05:52 beaker beakerd[19527]:  RuntimeError: <Server: beaker-recipe-11> failed to stop, status ACTIVE

Comment 3 matt jia 2016-08-02 00:34:36 UTC
I have increased the waiting time to 5 * 20 secs and it works in my environment.  We may need to use a big number for this in case the OpenStack server is too busy or slow..

On Gerrit:

    https://gerrit.beaker-project.org/#/c/5106/

Comment 6 Dan Callaghan 2017-02-21 18:48:11 UTC
Beaker 24.0 has been released.


Note You need to log in before you can comment on or make changes to this bug.