Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1694629 - [Regression] Server xxx failed to build "REVERT_RESIZE" test after upgrading from RHOS13 to RHOS14
Summary: [Regression] Server xxx failed to build "REVERT_RESIZE" test after upgrading ...
Keywords:
Status: CLOSED DUPLICATE of bug 1678681
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: nova-maint
QA Contact: nova-maint
URL:
Whiteboard: DFG:upgrades
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-01 09:52 UTC by Ronnie Rasouli
Modified: 2019-04-05 08:49 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-04 20:51:00 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Ronnie Rasouli 2019-04-01 09:52:34 UTC
Description of problem:

Tempest test failed:
 tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert

[tempest.api.compute.base] Failed to delete server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878

Failed to delete server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878
Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to build and is in ERROR status

Version-Release number of selected component (if applicable):
2019-03-18.1

How reproducible:
100%

Steps to Reproduce:
1. Deploy RHOS13
2. Upgrade undercloud and overcloud to RHOS14
3. Run tempest

Actual results:
failure on tempest tests

Expected results:
pass like it used to pass on previous builds

Additional info:

esponse - Headers: {'status': '200', u'content-length': '1729', 'content-location': 'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/v2.1/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878', u'x-compute-request-id': 'req-e5676815-2e24-43f9-8ae4-c14f5ae0b2ea', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Mon, 25 Mar 2019 16:41:49 GMT', u'content-type': 'application/json', u'x-openstack-request-id': 'req-e5676815-2e24-43f9-8ae4-c14f5ae0b2ea'}
        Body: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"tempest-ServerActionsTestJSON-187989746-network": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:4f:3a:ee", "version": 4, "addr": "10.100.0.10", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/v2.1/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878", "rel": "self"}, {"href": "https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878", "rel": "bookmark"}], "image": {"id": "769267a6-8db7-4b48-ad8f-400d8df155af", "links": [{"href": "https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/images/769267a6-8db7-4b48-ad8f-400d8df155af", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "error", "OS-SRV-USG:launched_at": "2019-03-25T16:36:36.000000", "flavor": {"id": "b2250279-2250-4c42-88a4-60bd95e4d764", "links": [{"href": "https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/flavors/b2250279-2250-4c42-88a4-60bd95e4d764", "rel": "bookmark"}]}, "id": "bfd1b2bd-2662-4aa4-97a0-e1a9adffb878", "security_groups": [{"name": "default"}], "user_id": "80d098d039b24ee7ab153cd7a6854972", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ERROR", "updated": "2019-03-25T16:41:48Z", "hostId": "7aeb5d4ca13e58c6b5cd23d4e39a3b2a7ad53de98cbb3d1fc3bd73e6", "OS-SRV-USG:terminated_at": null, "key_name": null, "name": "tempest-ServerActionsTestJSON-server-408964725", "created": "2019-03-25T16:35:58Z", "tenant_id": "bb640ac03b494b8fa9f5cad2d72c8ea1", "os-extended-volumes:volumes_attached": [], "fault": {"message": "Virtual Interface creation failed", "code": 500, "created": "2019-03-25T16:41:48Z"}, "metadata": {}}}
2019-03-25 12:41:50,440 795439 INFO     [tempest.common.waiters] State transition "REVERT_RESIZE/resize_reverting" ==> "ERROR/None" after 10 second wait
2019-03-25 12:41:50,440 795439 ERROR    [tempest.api.compute.base] Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to build and is in ERROR status
Details: {u'message': u'Virtual Interface creation failed', u'code': 500, u'created': u'2019-03-25T16:41:48Z'}
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base Traceback (most recent call last):
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 201, in server_check_teardown
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base     cls.server_id, 'ACTIVE')
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base     server_id=server_id)
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base BuildErrorException: Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to build and is in ERROR status
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base Details: {u'message': u'Virtual Interface creation failed', u'code': 500, u'created': u'2019-03-25T16:41:48Z'}
2019-03-25 12:41:50.440 795439 ERROR tempest.api.compute.base 
2019-03-25 12:41:50,756 795439 INFO     [tempest.lib.common.rest_client] Request (ServerActionsTestJSON:tearDown): 204 DELETE https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/v2.1/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 0.314s
2019-03-25 12:41:50,757 795439 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}

        Body: None
    Response - Headers: {'status': '404', u'content-length': '111', 'content-location': 'https://[2620:52:0:13b8:5054:ff:fe3e:1]:13774/v2.1/servers/bfd1b2bd-2662-4aa4-97a0-e1a9adffb878', u'x-compute-request-id': 'req-6a437388-506b-4d97-a601-472ae4a81b2a', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Mon, 25 Mar 2019 16:41:53 GMT', u'content-type': 'application/json; charset=UTF-8', u'x-openstack-request-id': 'req-6a437388-506b-4d97-a601-472ae4a81b2a'}
        Body: {"itemNotFound": {"message": "Instance bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 could not be found.", "code": 404}}
2019-03-25 12:41:53,249 795439 ERROR    [tempest.api.compute.base] Failed to delete server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base Traceback (most recent call last):
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 425, in delete_server
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     cls.servers_client.delete_server(server_id)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/servers_client.py", line 150, in delete_server
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     resp, body = self.delete("servers/%s" % server_id)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 310, in delete
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     return self.request('DELETE', url, extra_headers, headers, body)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     method, url, extra_headers, headers, body, chunked)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 670, in request
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     self._error_checker(resp, resp_body)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 776, in _error_checker
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base     raise exceptions.NotFound(resp_body, resp=resp)
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base NotFound: Object not found
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base Details: {u'message': u'Instance bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 could not be found.', u'code': 404}
2019-03-25 12:41:53.249 795439 ERROR tempest.api.compute.base
}}}

traceback-1: {{{
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_actions.py", line 66, in tearDown
    self.server_check_teardown()
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 201, in server_check_teardown
    cls.server_id, 'ACTIVE')
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to build and is in ERROR status
Details: {u'message': u'Virtual Interface creation failed', u'code': 500, u'created': u'2019-03-25T16:41:48Z'}
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_actions.py", line 370, in test_resize_server_revert
    waiters.wait_for_server_status(self.client, self.server_id, 'ACTIVE')
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 96, in wait_for_server_status
    raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (ServerActionsTestJSON:test_resize_server_revert) Server bfd1b2bd-2662-4aa4-97a0-e1a9adffb878 failed to reach ACTIVE status and task state "None" within the required time (300 s). Current status: REVERT_RESIZE. Current task state: resize_reverting.

Comment 2 Ronnie Rasouli 2019-04-01 10:13:54 UTC
Seems that the issue is also failing test: tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert_with_volume_attached[

Comment 3 Artom Lifshitz 2019-04-04 20:51:00 UTC
Given that I found the following in ./compute-0/var/log/containers/nova/nova-compute.log.1, I think we can safely close this as a duplicate of 1678681.

2019-03-25 16:41:47.956 1 WARNING nova.virt.libvirt.driver [req-41dcaefa-ad9f-4c07-8d8f-68583e060d6d 80d098d039b24ee7ab153cd7a6854972 bb640ac03b494b8fa9f5cad2d72c8ea1 - default default] [instance: bfd1b2bd-2662-
4aa4-97a0-e1a9adffb878] Timeout waiting for [('network-vif-plugged', u'f1b0f4aa-d018-4bed-a5e8-27a42c82d688')] for instance with vm_state resized and task_state resize_reverting.: Timeout: 300 seconds

and 

2019-03-25 16:41:48.287 1 ERROR nova.compute.manager [req-41dcaefa-ad9f-4c07-8d8f-68583e060d6d 80d098d039b24ee7ab153cd7a6854972 bb640ac03b494b8fa9f5cad2d72c8ea1 - default default] [instance: bfd1b2bd-2662-4aa4-97a0-e1a9adffb878] Setting instance vm_state to ERROR: VirtualInterfaceCreateException: Virtual Interface creation failed

*** This bug has been marked as a duplicate of bug 1678681 ***


Note You need to log in before you can comment on or make changes to this bug.